2023-11-16T23:29:13.951 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T23:29:13.951 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} --verbose --name choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi --archive /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178 --owner scheduled_choffman@teuthology -- /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/orig.config.yaml 2023-11-16T23:29:14.011 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T23:29:14.104 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178 branch: wip-choffman-62793 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} email: null first_in_suite: false job_id: '7461178' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-62793-testing sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6283 sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b sleep_before_teardown: 0 suite: fs suite_branch: wip-62793-testing suite_path: /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa suite_relpath: qa suite_repo: https://github.com/chrisphoffman/ceph.git suite_sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_21:22:00 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-16T23:29:14.104 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa; will attempt to use it 2023-11-16T23:29:14.106 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks 2023-11-16T23:29:14.107 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T23:29:14.109 INFO:teuthology.task.internal:Checking packages... 2023-11-16T23:29:14.133 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1c9565146c8647d4e00eb6c7ae626dc192680e7b' 2023-11-16T23:29:14.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T23:29:14.133 INFO:teuthology.packaging:ref: None 2023-11-16T23:29:14.133 INFO:teuthology.packaging:tag: None 2023-11-16T23:29:14.133 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-11-16T23:29:14.133 INFO:teuthology.packaging:sha1: 1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:29:14.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-11-16T23:29:14.300 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3906.g1c956514 2023-11-16T23:29:14.300 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T23:29:14.312 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T23:29:14.312 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T23:29:14.321 INFO:teuthology.task.internal:Saving configuration 2023-11-16T23:29:14.354 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T23:29:14.360 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T23:29:14.387 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.704348', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T23:29:14.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.708576', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T23:29:14.440 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.706881', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T23:29:14.440 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T23:29:14.446 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-11-16T23:29:14.446 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-11-16T23:29:14.446 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-11-16T23:29:14.446 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T23:29:14.581 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff2e74a3f70>, signals=[15]) 2023-11-16T23:29:14.582 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T23:29:14.588 INFO:teuthology.task.internal:Opening connections... 2023-11-16T23:29:14.588 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2023-11-16T23:29:14.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:29:14.657 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-11-16T23:29:14.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:29:14.733 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2023-11-16T23:29:14.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:29:14.803 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T23:29:14.810 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2023-11-16T23:29:14.832 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2023-11-16T23:29:14.832 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2023-11-16T23:29:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T23:29:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T23:29:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:ID="rhel" 2023-11-16T23:29:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="fedora" 2023-11-16T23:29:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8.6" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T23:29:14.892 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T23:29:14.893 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2023-11-16T23:29:14.923 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-11-16T23:29:14.945 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-11-16T23:29:14.946 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-11-16T23:29:15.004 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T23:29:15.004 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:ID="rhel" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="fedora" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8.6" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:29:15.005 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T23:29:15.006 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T23:29:15.006 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T23:29:15.006 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T23:29:15.006 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-11-16T23:29:15.029 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2023-11-16T23:29:15.051 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2023-11-16T23:29:15.051 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2023-11-16T23:29:15.111 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T23:29:15.111 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T23:29:15.111 INFO:teuthology.orchestra.run.smithi173.stdout:ID="rhel" 2023-11-16T23:29:15.111 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="fedora" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8.6" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T23:29:15.113 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T23:29:15.114 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T23:29:15.114 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2023-11-16T23:29:15.137 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T23:29:15.148 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T23:29:15.153 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T23:29:15.153 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2023-11-16T23:29:15.155 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-11-16T23:29:15.159 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2023-11-16T23:29:15.179 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T23:29:15.184 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T23:29:15.184 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2023-11-16T23:29:15.215 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-11-16T23:29:15.217 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2023-11-16T23:29:15.247 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T23:29:15.326 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T23:29:15.351 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-11-16T23:29:15.352 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-11-16T23:29:15.352 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-11-16T23:29:15.352 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T23:29:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:29:15.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-11-16T23:29:15.625 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T23:29:15.625 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5b73517e0e2fa132ea79290e5fc06ea5349b2288"... 2023-11-16T23:29:15.625 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-11-16T23:29:15.646 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-16T23:29:15.646 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T23:29:15.646 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-16T23:29:15.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum upgrade -y linux-firmware 2023-11-16T23:29:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-11-16T23:29:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2023-11-16T23:29:18.502 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-16T23:29:19.136 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-11-16T23:29:22.878 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 550 kB/s | 24 kB 00:00 2023-11-16T23:29:23.787 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-11-16T23:29:23.788 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-11-16T23:29:23.789 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-11-16T23:29:23.924 INFO:teuthology.task.kernel:Downloading kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T23:29:23.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T23:29:24.071 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ 2023-11-16T23:29:24.071 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ --input-file=- 2023-11-16T23:29:24.074 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi023.front.sepia.ceph.com to complete... 2023-11-16T23:29:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16 23:29:37 URL:https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm [383129572/383129572] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-16T23:29:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:FINISHED --2023-11-16 23:29:37-- 2023-11-16T23:29:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:Total wall clock time: 13s 2023-11-16T23:29:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:Downloaded: 1 files, 365M in 13s (28.0 MB/s) 2023-11-16T23:29:37.473 INFO:teuthology.task.kernel:Installing kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T23:29:37.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-16T23:29:39.133 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying... ######################################## 2023-11-16T23:29:39.191 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing... ######################################## 2023-11-16T23:29:39.271 INFO:teuthology.orchestra.run.smithi023.stdout:Updating / installing... 2023-11-16T23:30:20.221 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-6.6.0_rc5_g5b73517e0e2f-1 ######################################## 2023-11-16T23:30:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-16T23:30:44.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-16T23:30:44.383 DEBUG:teuthology.orchestra.run.smithi023:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/config-6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-16T23:30:44.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-16T23:30:44.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm.ko 2023-11-16T23:30:44.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ansi_cprng.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/anubis.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/arc4.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_memcpy.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_pq.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_tx.ko 2023-11-16T23:30:44.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_xor.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/raid6test.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authenc.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authencesn.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blake2b_generic.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_common.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_generic.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/camellia_generic.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast5_generic.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast6_generic.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast_common.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ccm.ko 2023-11-16T23:30:44.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cmac.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crc32_generic.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_engine.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_user.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cts.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/des_generic.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/dh_generic.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/echainiv.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/essiv.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/fcrypt.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/khazad.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4hc.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/md4.ko 2023-11-16T23:30:44.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/michael_mic.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcbc.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcrypt.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/rmd160.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/seed.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/serpent_generic.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tcrypt.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tea.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_common.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_generic.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/vmac.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/wp512.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xcbc.ko 2023-11-16T23:30:44.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xor.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xxhash_generic.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_pad.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/custom_method.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/ec_sys.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/platform_profile.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbs.ko 2023-11-16T23:30:44.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbshc.ko 2023-11-16T23:30:44.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/video.ko 2023-11-16T23:30:44.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata 2023-11-16T23:30:44.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/acard-ahci.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ahci_platform.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ata_generic.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/libahci_platform.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_acpi.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ali.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_amd.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_artop.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atiixp.ko 2023-11-16T23:30:44.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atp867x.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd640.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd64x.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cypress.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_efar.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt366.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt37x.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it8213.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it821x.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_jmicron.ko 2023-11-16T23:30:44.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_marvell.ko 2023-11-16T23:30:44.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_mpiix.ko 2023-11-16T23:30:44.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_netcell.ko 2023-11-16T23:30:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ninja32.ko 2023-11-16T23:30:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87410.ko 2023-11-16T23:30:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87415.ko 2023-11-16T23:30:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_oldpiix.ko 2023-11-16T23:30:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_opti.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_optidma.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pcmcia.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_piccolo.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_rdc.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sch.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_serverworks.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sil680.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sis.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sl82c105.ko 2023-11-16T23:30:44.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_triflex.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_via.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pdc_adma.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_inic162x.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_mv.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_nv.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_promise.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_qstor.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil24.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sis.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_svw.ko 2023-11-16T23:30:44.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sx4.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_uli.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_via.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_vsc.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay/ks0108.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma 2023-11-16T23:30:44.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma/bcma.ko 2023-11-16T23:30:44.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block 2023-11-16T23:30:44.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe/aoe.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/brd.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd/drbd.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/floppy.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/loop.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/nbd.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk/null_blk.ko 2023-11-16T23:30:44.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/pktcdvd.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/rbd.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/virtio_blk.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkfront.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram/zram.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hangcheck-timer.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-16T23:30:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/via-rng.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-16T23:30:44.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/lp.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave/mwave.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ppdev.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tlclk.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-16T23:30:44.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/uv_mmtimer.ko 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/virtio_console.ko 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq 2023-11-16T23:30:44.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-16T23:30:44.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-16T23:30:44.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-16T23:30:44.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-16T23:30:44.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-16T23:30:44.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-aes.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-sha.ko 2023-11-16T23:30:44.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca/dca.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/amd64_edac.ko 2023-11-16T23:30:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/e752x_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/edac_mce_amd.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3000_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3200_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5100_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5400_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7300_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7core_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i82975x_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/ie31200_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/sb_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/x38_edac.ko 2023-11-16T23:30:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/edd.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-ich.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-appleir.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-aureal.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-axff.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-dr.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elecom.ko 2023-11-16T23:30:44.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elo.ko 2023-11-16T23:30:44.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-emsff.ko 2023-11-16T23:30:44.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gaff.ko 2023-11-16T23:30:44.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gt683r.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gyration.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtekff.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-hyperv.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-icade.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-keytouch.ko 2023-11-16T23:30:44.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-kye.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lcpower.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-led.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lenovo.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-multitouch.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-ortek.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-penmount.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-petalynx.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-picolcd.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-pl.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-plantronics.ko 2023-11-16T23:30:44.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-primax.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-rmi.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-common.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-16T23:30:44.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-saitek.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-samsung.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sjoy.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sony.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-speedlink.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-steelseries.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sunplus.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tivo.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tmff.ko 2023-11-16T23:30:44.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-topseed.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-twinhan.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-uclogic.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-waltop.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-wiimote.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-xinmo.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zpff.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zydacron.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/uhid.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/wacom.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_balloon.ko 2023-11-16T23:30:44.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_utils.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_vmbus.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru3.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7414.ko 2023-11-16T23:30:44.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7418.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adc128d818.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1025.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1026.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1029.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1031.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm9240.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ads7828.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7410.ko 2023-11-16T23:30:44.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7411.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7462.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7470.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7475.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7x10.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/amc6821.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/applesmc.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asb100.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asc7621.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-16T23:30:44.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/atxp1.ko 2023-11-16T23:30:44.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/coretemp.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dme1737.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds1621.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds620.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc1403.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc6w201.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71805f.ko 2023-11-16T23:30:44.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71882fg.ko 2023-11-16T23:30:44.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f75375s.ko 2023-11-16T23:30:44.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fam15h_power.ko 2023-11-16T23:30:44.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fschmd.ko 2023-11-16T23:30:44.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g760a.ko 2023-11-16T23:30:44.511 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g762.ko 2023-11-16T23:30:44.511 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl518sm.ko 2023-11-16T23:30:44.511 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl520sm.ko 2023-11-16T23:30:44.511 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5500_temp.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5k_amb.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmaem.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmpex.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina209.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina2xx.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/it87.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k10temp.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k8temp.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lineage-pem.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm63.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm73.ko 2023-11-16T23:30:44.512 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm75.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm77.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm78.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm80.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm83.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm85.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm87.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm90.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm92.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm93.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95234.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95241.ko 2023-11-16T23:30:44.513 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95245.ko 2023-11-16T23:30:44.514 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc2945.ko 2023-11-16T23:30:44.514 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4151.ko 2023-11-16T23:30:44.514 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4215.ko 2023-11-16T23:30:44.514 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4222.ko 2023-11-16T23:30:44.519 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4245.ko 2023-11-16T23:30:44.519 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4260.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4261.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max16065.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1619.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1668.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max197.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6639.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6650.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6697.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/mcp3021.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6683.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775-core.ko 2023-11-16T23:30:44.520 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct7802.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87360.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87427.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pcf8591.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-16T23:30:44.521 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/powr1220.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5627.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5636.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht15.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht21.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/shtc1.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sis5595.ko 2023-11-16T23:30:44.522 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47b397.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m1.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m192.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/thmc50.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp102.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp103.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp401.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp421.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via-cputemp.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via686a.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt1211.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt8231.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627ehf.ko 2023-11-16T23:30:44.523 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627hf.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83781d.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83791d.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83792d.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83793.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83795.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l785ts.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l786ng.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses 2023-11-16T23:30:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-16T23:30:44.525 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-dev.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-mux.ko 2023-11-16T23:30:44.526 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-smbus.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-stub.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_core.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-16T23:30:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser 2023-11-16T23:30:44.528 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/gameport.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/joydev.ko 2023-11-16T23:30:44.529 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/appletouch.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/bcm5974.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/cyapatp.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/sermouse.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-16T23:30:44.530 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio 2023-11-16T23:30:44.531 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/altera_ps2.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/arc_ps2.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/serio_raw.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/sparse-keymap.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-blinkm.ko 2023-11-16T23:30:44.532 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lm3530.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lp3944.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-ss4200.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-16T23:30:44.533 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache/bcache.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-bio-prison.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache-smq.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-crypt.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-delay.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-flakey.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-log-userspace.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-multipath.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-queue-length.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-raid.ko 2023-11-16T23:30:44.534 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-round-robin.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-service-time.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-switch.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-thin-pool.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-verity.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/faulty.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/linear.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/multipath.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid0.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid1.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid10.ko 2023-11-16T23:30:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid456.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptbase.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptctl.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptfc.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptlan.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptsas.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptscsih.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptspi.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_ich.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_sch.ko 2023-11-16T23:30:44.536 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/mfd-core.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/sm501.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/viperboard.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/vx855.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/wl1273-core.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds9802als.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds990x.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/bh1770glc.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710/cb710.ko 2023-11-16T23:30:44.537 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/at24.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/max6875.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/enclosure.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/hpilo.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29003.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29020.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-16T23:30:44.538 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-me.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-txe.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_7xx1.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_core.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tsl2550.ko 2023-11-16T23:30:44.539 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_balloon.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding/bonding.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-16T23:30:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dummy.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/eql.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec 2023-11-16T23:30:44.541 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-16T23:30:44.542 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-16T23:30:44.543 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-16T23:30:44.544 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic 2023-11-16T23:30:44.545 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dnet.ko 2023-11-16T23:30:44.546 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ethoc.ko 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/fealnx.ko 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel 2023-11-16T23:30:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-16T23:30:44.585 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-16T23:30:44.586 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/jme.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-16T23:30:44.587 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-16T23:30:44.588 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-16T23:30:44.589 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-16T23:30:44.590 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-16T23:30:44.591 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-16T23:30:44.592 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/geneve.ko 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ifb.ko 2023-11-16T23:30:44.593 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvlan.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvtap.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mii.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/net_failover.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/netconsole.ko 2023-11-16T23:30:44.636 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/nlmon.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/amd.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm87xx.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/broadcom.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/cicada.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/davicom.ko 2023-11-16T23:30:44.637 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/dp83640.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/et1011c.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/icplus.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/lxt.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/marvell.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/micrel.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/national.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/phylink.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/qsemi.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/realtek.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/smsc.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/ste10Xp.ko 2023-11-16T23:30:44.638 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/vitesse.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_async.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppoe.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppox.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pptp.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slhc.ko 2023-11-16T23:30:44.639 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slip.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/sungem_phy.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tap.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_random.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tun.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/veth.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/virtio_net.ko 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3 2023-11-16T23:30:44.640 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan/vxlan.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netfront.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb/ntb.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme-core.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme.ko 2023-11-16T23:30:44.641 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_cs.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_pc.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_serial.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/xen-pcifront.ko 2023-11-16T23:30:44.642 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/i82092.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/pd6729.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acerhdf.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-16T23:30:44.643 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/hdaps.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-16T23:30:44.644 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel_ips.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-16T23:30:44.645 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps_parport.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/pps_core.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp_kvm.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-16T23:30:44.646 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-em3027.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-16T23:30:44.647 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-max6900.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-16T23:30:44.648 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-x1205.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-9xxx.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-sas.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-xxxx.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/BusLogic.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/a100u2w.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/advansys.ko 2023-11-16T23:30:44.649 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/am53c974.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/atp870u.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa/bfa.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-16T23:30:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ch.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dc395x.ko 2023-11-16T23:30:44.651 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dmx3191d.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esp_scsi.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic 2023-11-16T23:30:44.652 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic/fnic.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hpsa.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hptiop.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hv_storvsc.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/initio.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ipr.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ips.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci/isci.ko 2023-11-16T23:30:44.662 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc/libfc.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas/libsas.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-16T23:30:44.663 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvumi.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pmcraid.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla1280.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-16T23:30:44.664 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/raid_class.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_debug.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ses.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/st.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/stex.ko 2023-11-16T23:30:44.665 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/virtio_scsi.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/wd719x.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb/ssb.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-16T23:30:44.666 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_file.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_iblock.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_mod.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_pscsi.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_user.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-16T23:30:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty 2023-11-16T23:30:44.668 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_gsm.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_hdlc.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/nozomi.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/arc_uart.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core 2023-11-16T23:30:44.669 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_aec.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_cif.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_pci_generic.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_sercos3.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-acm.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usblp.ko 2023-11-16T23:30:44.670 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usbtmc.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/mdc800.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/microtek.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/adutux.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/appledisplay.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi26.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi62.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ezusb.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/idmouse.ko 2023-11-16T23:30:44.671 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/iowarrior.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ldusb.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/legousbtower.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usb3503.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usblcd.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/uss720.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/yurex.ko 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy 2023-11-16T23:30:44.672 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy/phy-generic.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/aircable.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ark3116.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ch341.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cp210x.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cyberjack.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/empeg.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-16T23:30:44.673 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_ti.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipaq.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipw.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ir-usb.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mct_u232.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7720.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7840.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/navman.ko 2023-11-16T23:30:44.674 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/omninet.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/opticon.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/option.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/oti6858.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/pl2303.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcaux.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcserial.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/quatech2.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/safe_serial.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/sierra.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ssu100.ko 2023-11-16T23:30:44.675 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/symbolserial.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_debug.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/visor.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/whiteheat.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/uas.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-16T23:30:44.676 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-karma.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/usb-storage.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-16T23:30:44.677 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_net.ko 2023-11-16T23:30:44.678 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_scsi.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/apple_bl.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/lcd.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_balloon.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_mmio.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_ring.ko 2023-11-16T23:30:44.679 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2482.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2490.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-16T23:30:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/wire.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/hpwdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/i6300esb.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ib700wdt.ko 2023-11-16T23:30:44.681 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ibmasr.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it87_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/machzwd.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/nv_tco.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/softdog.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/via_wdt.ko 2023-11-16T23:30:44.682 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/wdt_pci.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/xen_wdt.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-evtchn.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntalloc.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntdev.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-privcmd.ko 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs 2023-11-16T23:30:44.683 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p/9p.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs/affs.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs/befs.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/binfmt_misc.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs/btrfs.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles/cachefiles.ko 2023-11-16T23:30:44.684 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph/ceph.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda/coda.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs/cramfs.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm/dlm.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs/f2fs.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/fat.ko 2023-11-16T23:30:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/msdos.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/vfat.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache/fscache.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/cuse.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/fuse.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2/gfs2.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs/hfs.ko 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus 2023-11-16T23:30:44.686 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus/hfsplus.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs/isofs.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs/jfs.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd/lockd.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix/minix.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs/netfs.ko 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout 2023-11-16T23:30:44.687 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfs.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv3.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv4.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/grace.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/nfs_acl.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd/nfsd.ko 2023-11-16T23:30:44.688 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2/nilfs2.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-celtic.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-centeuro.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-croatian.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-cyrillic.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-gaelic.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-greek.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-iceland.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-inuit.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-roman.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-romanian.ko 2023-11-16T23:30:44.689 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-turkish.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1250.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1251.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1255.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp737.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp775.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp850.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp852.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp855.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp857.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp860.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp861.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp862.ko 2023-11-16T23:30:44.690 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp863.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp864.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp865.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp866.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp869.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp874.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp932.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp936.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp949.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp950.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_euc-jp.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-1.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-13.ko 2023-11-16T23:30:44.691 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-14.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-15.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-2.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-3.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-4.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-5.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-6.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-7.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-9.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-r.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-ru.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-u.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_utf8.ko 2023-11-16T23:30:44.692 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs 2023-11-16T23:30:44.693 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs/overlay.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore/ramoops.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs/reiserfs.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs/romfs.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client/cifs.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_arc4.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_md4.ko 2023-11-16T23:30:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs/squashfs.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv/sysv.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf/udf.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs/ufs.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs/xfs.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu/rcutorture.ko 2023-11-16T23:30:44.695 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/torture.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/asn1_encoder.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc-itu-t.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc8.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libarc4.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libdes.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/libcrc32c.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lru_cache.ko 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4 2023-11-16T23:30:44.696 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4_compress.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4hc_compress.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math/cordic.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6/raid6_pq.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_bm.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_fsm.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_kmp.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd/zstd_compress.ko 2023-11-16T23:30:44.697 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm/hwpoison-inject.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/6lowpan.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_dest.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_fragment.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_hop.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_mobility.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_routing.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_udp.ko 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802 2023-11-16T23:30:44.698 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/garp.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/mrp.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/p8022.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/psnap.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/stp.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q/8021q.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_fd.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_rdma.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_virtio.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv/batman-adv.ko 2023-11-16T23:30:44.699 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/br_netfilter.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/bridge.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-16T23:30:44.700 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtables.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-16T23:30:44.701 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph/libceph.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/failover.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/pktgen.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_diag.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv4.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv6.ko 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver 2023-11-16T23:30:44.702 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver/dns_resolver.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/dsa_core.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_brcm.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_dsa.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_none.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_trailer.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4 2023-11-16T23:30:44.703 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ah4.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/esp4.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/fou.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/gre.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/inet_diag.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_gre.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_tunnel.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_vti.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipcomp.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipip.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-16T23:30:44.704 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-16T23:30:44.705 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_bic.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_dctcp.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_diag.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_highspeed.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_htcp.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_hybla.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_illinois.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_lp.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_scalable.ko 2023-11-16T23:30:44.706 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_vegas.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_veno.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_westwood.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_yeah.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tunnel4.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_diag.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_tunnel.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ah6.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/esp6.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/fou6.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_tunnel.ko 2023-11-16T23:30:44.707 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_vti.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ipcomp6.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-16T23:30:44.708 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-16T23:30:44.709 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/sit.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/tunnel6.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key/af_key.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_core.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_eth.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip6.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_netlink.ko 2023-11-16T23:30:44.710 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ppp.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc/llc.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154/mac802154.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls/mpls_gso.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-16T23:30:44.711 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-16T23:30:44.712 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-16T23:30:44.713 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conncount.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-16T23:30:44.714 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_log_syslog.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_irc.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_sip.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-16T23:30:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_tables.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_log.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_chain_nat.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_compat.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_ct.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fib.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_hash.ko 2023-11-16T23:30:44.716 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_limit.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_log.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_masq.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_nat.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_numgen.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_queue.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_quota.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_redir.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject_inet.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_socket.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_tunnel.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_AUDIT.ko 2023-11-16T23:30:44.717 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CT.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_DSCP.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HL.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HMARK.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LED.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LOG.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NETMAP.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFLOG.ko 2023-11-16T23:30:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_RATEEST.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_SECMARK.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TEE.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TPROXY.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TRACE.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_addrtype.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_bpf.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cgroup.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cluster.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_comment.ko 2023-11-16T23:30:44.719 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connbytes.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlabel.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlimit.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connmark.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_conntrack.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cpu.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dccp.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_devgroup.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dscp.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ecn.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_esp.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hashlimit.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_helper.ko 2023-11-16T23:30:44.720 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hl.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipcomp.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_iprange.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipvs.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_l2tp.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_length.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_limit.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mac.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mark.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_multiport.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nat.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nfacct.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_osf.ko 2023-11-16T23:30:44.721 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_owner.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_physdev.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_pkttype.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_policy.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_quota.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_rateest.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_realm.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_recent.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_sctp.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_set.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_socket.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_state.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_statistic.ko 2023-11-16T23:30:44.722 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_string.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_tcpmss.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_time.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_u32.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink/netlink_diag.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh/nsh.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/openvswitch.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-geneve.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-gre.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-vxlan.ko 2023-11-16T23:30:44.723 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet/af_packet_diag.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_rdma.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_tcp.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_csum.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_gact.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_ipt.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_mirred.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_nat.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_pedit.ko 2023-11-16T23:30:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_police.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_simple.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_skbedit.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_vlan.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_basic.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_bpf.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_flow.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_fw.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_route.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_u32.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_cmp.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_ipset.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_meta.ko 2023-11-16T23:30:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_nbyte.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_text.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_u32.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_choke.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_codel.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_drr.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_fq.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_gred.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hfsc.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hhf.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_htb.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_ingress.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio.ko 2023-11-16T23:30:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio_lib.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_multiq.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_netem.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_pie.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_plug.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_prio.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_qfq.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_red.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfb.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfq.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_tbf.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_teql.ko 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp 2023-11-16T23:30:44.727 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp_diag.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/sunrpc.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/diag.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/tipc.ko 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix 2023-11-16T23:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix/unix_diag.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys 2023-11-16T23:30:44.729 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys/trusted.ko 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib/irqbypass.ko 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias.bin 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.bin 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.modinfo 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep.bin 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.devname 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.order 2023-11-16T23:30:44.730 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.softdep 2023-11-16T23:30:44.731 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols 2023-11-16T23:30:44.731 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols.bin 2023-11-16T23:30:44.731 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.731 DEBUG:teuthology.orchestra.run.smithi023:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:postinstall scriptlet (using /bin/sh): 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f -a -r /boot/System.map-6.6.0-rc5-g5b73517e0e2f ]; then 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:cp /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:cp /boot/System.map-6.6.0-rc5-g5b73517e0e2f /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/installkernel 6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T23:30:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:new-kernel-pkg --remove 6.6.0-rc5-g5b73517e0e2f --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g5b73517e0e2f.img 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-install remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-11-16T23:30:44.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.02 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 123.el8 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Group : System Environment/Base 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 9265450 2023-11-16T23:30:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Relocations : (not relocatable) 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : Red Hat, Inc. 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : Red Hat, Inc. 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T23:30:44.843 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T23:30:44.844 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2023-11-16T23:30:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:30:44.844 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T23:30:44.845 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T23:30:44.845 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:44.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T23:30:49.054 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2023-11-16T23:30:51.194 INFO:teuthology.orchestra.run.smithi023.stderr:done 2023-11-16T23:30:51.196 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2023-11-16T23:30:51.214 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.nQzzUPwV8p 2023-11-16T23:30:51.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nQzzUPwV8p 2023-11-16T23:30:51.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.nQzzUPwV8p 2023-11-16T23:30:51.412 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.nQzzUPwV8p is 6KB 2023-11-16T23:30:51.465 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.nQzzUPwV8p 2023-11-16T23:30:51.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T23:30:51.548 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-11-16T23:30:51.548 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-11-16T23:30:51.548 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f.conf 2023-11-16T23:30:51.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:30:51.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2023-11-16T23:31:21.705 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-11-16T23:31:21.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:31:21.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:31:40.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-11-16T23:31:49.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:31:49.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:31:52.090 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-11-16T23:32:04.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:32:04.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:32:07.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-11-16T23:32:22.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:32:22.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:33:22.171 DEBUG:teuthology.orchestra.remote:timed out 2023-11-16T23:33:40.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:33:40.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:33:40.536 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-11-16T23:33:40.941 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:33:40.941 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g5b73517e0e2f"... 2023-11-16T23:33:40.941 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-11-16T23:33:40.986 INFO:teuthology.orchestra.run.smithi023.stdout:6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:33:40.986 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g5b73517e0e2f vs 6.6.0-rc5-g5b73517e0e2f 2023-11-16T23:33:40.987 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T23:33:40.987 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-11-16T23:33:41.988 DEBUG:teuthology.parallel:result is None 2023-11-16T23:33:41.988 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T23:33:42.003 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T23:33:42.003 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T23:33:42.008 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T23:33:42.011 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T23:33:42.051 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T23:33:42.057 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T23:33:42.105 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T23:33:42.106 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T23:33:42.117 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T23:33:42.119 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T23:33:42.169 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T23:33:42.177 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T23:33:42.178 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T23:33:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:33:42.212 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T23:33:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:33:42.231 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T23:33:42.235 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T23:33:42.254 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T23:33:42.326 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.342 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.357 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.372 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.399 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T23:33:42.402 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T23:33:42.409 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T23:33:42.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T23:33:42.417 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T23:33:42.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T23:33:42.482 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T23:33:42.491 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T23:33:42.491 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T23:33:42.496 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T23:33:42.498 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T23:33:42.535 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:42.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:42.712 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:42.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:42.850 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:33:42.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T23:33:42.938 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:42.981 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:43.044 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:43.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:43.179 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:33:43.179 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T23:33:43.245 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:43.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T23:33:43.354 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:43.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T23:33:43.442 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:33:43.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T23:33:43.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2023-11-16T23:33:43.516 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-11-16T23:33:43.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2023-11-16T23:33:43.585 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T23:33:43.613 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T23:33:43.628 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T23:33:44.119 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T23:33:44.127 INFO:teuthology.task.internal:Starting timer... 2023-11-16T23:33:44.128 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T23:33:44.138 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T23:33:44.147 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T23:33:44.147 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2023-11-16T23:33:44.272 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2023-11-16T23:33:44.275 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2023-11-16T23:33:44.331 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2023-11-16T23:33:44.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2023-11-16T23:33:44.402 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2023-11-16T23:33:44.404 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T23:33:44.404 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2023-11-16T23:33:44.454 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2023-11-16T23:33:44.454 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2023-11-16T23:33:44.480 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2023-11-16T23:33:44.480 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2023-11-16T23:33:44.501 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2023-11-16T23:33:44.501 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2023-11-16T23:33:44.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T23:33:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:33:44.583 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T23:33:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:33:44.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T23:33:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T23:33:44.638 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T23:33:44.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2023-11-16T23:33:44.711 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2023-11-16T23:33:44.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2023-11-16T23:33:44.766 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T23:33:44.776 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-16T23:33:44.782 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-16T23:33:44.940 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T23:33:44.957 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T23:33:44.958 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi156.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T23:46:35.661 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2023-11-16T23:46:35.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:46:35.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:46:35.756 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-11-16T23:46:35.868 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-11-16T23:46:35.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-16T23:46:35.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:46:35.937 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-11-16T23:46:36.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-16T23:46:36.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-16T23:46:36.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T23:46:36.084 DEBUG:teuthology.orchestra.run.smithi173:> true 2023-11-16T23:46:36.155 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-16T23:46:36.155 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T23:46:36.166 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T23:46:36.166 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T23:46:36.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T23:46:36.171 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T23:46:36.171 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T23:46:36.175 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T23:46:36.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T23:46:36.207 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T23:46:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T23:46:36.228 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T23:46:36.242 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T23:46:36.243 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T23:46:36.278 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T23:46:36.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2023-11-16T23:46:36.325 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2023-11-16T23:46:36.327 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2023-11-16T23:46:36.338 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2023-11-16T23:46:36.342 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T23:46:36.353 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T23:46:36.356 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T23:46:36.368 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T23:46:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2023-11-16T23:46:36.401 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2023-11-16T23:46:36.434 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2023-11-16T23:46:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T23:46:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2023-11-16T23:46:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.438 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T23:46:36.455 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-11-16T23:46:36.455 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T23:46:36.456 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-11-16T23:46:36.456 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.456 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.456 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.456 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.474 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T23:46:36.570 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.577 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T23:46:36.578 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T23:46:36.590 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T23:46:36.591 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2023-11-16T23:46:36.591 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T23:46:36.591 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi156.front.sepia.ceph.com 2023-11-16T23:46:36.592 DEBUG:teuthology.orchestra.run.smithi156:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T23:46:36.592 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2023-11-16T23:46:36.593 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:36.975 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:37.122 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:37.799 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:04:58 ago on Thu 16 Nov 2023 11:41:39 PM UTC. 2023-11-16T23:46:37.819 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:05:00 ago on Thu 16 Nov 2023 11:41:37 PM UTC. 2023-11-16T23:46:38.144 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:50 ago on Thu 16 Nov 2023 11:44:47 PM UTC. 2023-11-16T23:46:39.535 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:Resetting modules: 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-16T23:46:39.537 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:46:39.680 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout:Resetting modules: 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-11-16T23:46:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:46:40.191 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout:Resetting modules: 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-11-16T23:46:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:46:40.520 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:05:03 ago on Thu 16 Nov 2023 11:41:37 PM UTC. 2023-11-16T23:46:40.640 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:05:01 ago on Thu 16 Nov 2023 11:41:39 PM UTC. 2023-11-16T23:46:41.491 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:53 ago on Thu 16 Nov 2023 11:44:47 PM UTC. 2023-11-16T23:46:43.106 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-11-16T23:46:43.347 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-11-16T23:46:43.584 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:43.825 INFO:teuthology.orchestra.run.smithi173.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:44.111 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-11-16T23:46:44.422 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:05:06 ago on Thu 16 Nov 2023 11:41:37 PM UTC. 2023-11-16T23:46:44.661 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:05:05 ago on Thu 16 Nov 2023 11:41:39 PM UTC. 2023-11-16T23:46:44.750 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-11-16T23:46:45.817 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:58 ago on Thu 16 Nov 2023 11:44:47 PM UTC. 2023-11-16T23:46:46.159 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-11-16T23:46:46.252 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-11-16T23:46:46.252 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-11-16T23:46:46.252 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-11-16T23:46:46.252 INFO:teuthology.orchestra.run.smithi156.stdout:Installing group/module packages: 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout:Installing module profiles: 2023-11-16T23:46:46.253 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools/common 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools rhel8 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout:Install 10 Packages 2023-11-16T23:46:46.254 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:46:46.255 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 18 M 2023-11-16T23:46:46.255 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 65 M 2023-11-16T23:46:46.255 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-11-16T23:46:46.350 INFO:teuthology.orchestra.run.smithi156.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 991 kB/s | 94 kB 00:00 2023-11-16T23:46:46.375 INFO:teuthology.orchestra.run.smithi156.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 207 kB/s | 25 kB 00:00 2023-11-16T23:46:46.384 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-11-16T23:46:46.400 INFO:teuthology.orchestra.run.smithi156.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 281 kB/s | 41 kB 00:00 2023-11-16T23:46:46.425 INFO:teuthology.orchestra.run.smithi156.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 656 kB/s | 49 kB 00:00 2023-11-16T23:46:46.467 INFO:teuthology.orchestra.run.smithi156.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.6 MB/s | 524 kB 00:00 2023-11-16T23:46:46.478 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================================ 2023-11-16T23:46:46.478 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-11-16T23:46:46.478 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================================ 2023-11-16T23:46:46.478 INFO:teuthology.orchestra.run.smithi173.stdout:Installing group/module packages: 2023-11-16T23:46:46.478 INFO:teuthology.orchestra.run.smithi173.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T23:46:46.479 INFO:teuthology.orchestra.run.smithi173.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout:Installing module profiles: 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools/common 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout:Enabling module streams: 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools rhel8 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================================ 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout:Install 10 Packages 2023-11-16T23:46:46.480 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:46:46.481 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 18 M 2023-11-16T23:46:46.482 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 65 M 2023-11-16T23:46:46.482 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-11-16T23:46:46.493 INFO:teuthology.orchestra.run.smithi156.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.2 MB/s | 149 kB 00:00 2023-11-16T23:46:46.548 INFO:teuthology.orchestra.run.smithi156.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.6 MB/s | 213 kB 00:00 2023-11-16T23:46:46.552 INFO:teuthology.orchestra.run.smithi173.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.3 MB/s | 94 kB 00:00 2023-11-16T23:46:46.577 INFO:teuthology.orchestra.run.smithi173.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 257 kB/s | 25 kB 00:00 2023-11-16T23:46:46.603 INFO:teuthology.orchestra.run.smithi173.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 335 kB/s | 41 kB 00:00 2023-11-16T23:46:46.663 INFO:teuthology.orchestra.run.smithi173.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 449 kB/s | 49 kB 00:00 2023-11-16T23:46:46.668 INFO:teuthology.orchestra.run.smithi156.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 18 MB/s | 2.2 MB 00:00 2023-11-16T23:46:46.703 INFO:teuthology.orchestra.run.smithi173.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.1 MB/s | 524 kB 00:00 2023-11-16T23:46:46.728 INFO:teuthology.orchestra.run.smithi173.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.2 MB/s | 149 kB 00:00 2023-11-16T23:46:46.753 INFO:teuthology.orchestra.run.smithi173.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-11-16T23:46:46.768 INFO:teuthology.orchestra.run.smithi156.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-11-16T23:46:46.846 INFO:teuthology.orchestra.run.smithi173.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-11-16T23:46:46.852 INFO:teuthology.orchestra.run.smithi156.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-11-16T23:46:46.852 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:46:46.852 INFO:teuthology.orchestra.run.smithi156.stdout:Total 30 MB/s | 18 MB 00:00 2023-11-16T23:46:46.962 INFO:teuthology.orchestra.run.smithi173.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 30 MB/s | 7.0 MB 00:00 2023-11-16T23:46:47.054 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-11-16T23:46:47.073 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-11-16T23:46:47.073 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-11-16T23:46:47.084 INFO:teuthology.orchestra.run.smithi173.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 17 MB/s | 8.0 MB 00:00 2023-11-16T23:46:47.084 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:46:47.084 INFO:teuthology.orchestra.run.smithi173.stdout:Total 30 MB/s | 18 MB 00:00 2023-11-16T23:46:47.254 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-11-16T23:46:47.258 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-11-16T23:46:47.289 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-11-16T23:46:47.308 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-11-16T23:46:47.308 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-11-16T23:46:47.489 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-11-16T23:46:47.493 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-11-16T23:46:47.786 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-11-16T23:46:47.943 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-11-16T23:46:47.959 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T23:46:48.004 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================================ 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================================ 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout:Installing group/module packages: 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T23:46:48.049 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout:Installing module profiles: 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools/common 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools rhel8 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================================ 2023-11-16T23:46:48.050 INFO:teuthology.orchestra.run.smithi023.stdout:Install 10 Packages 2023-11-16T23:46:48.051 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:46:48.051 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 18 M 2023-11-16T23:46:48.051 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 65 M 2023-11-16T23:46:48.051 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-11-16T23:46:48.125 INFO:teuthology.orchestra.run.smithi023.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 560 kB/s | 41 kB 00:00 2023-11-16T23:46:48.150 INFO:teuthology.orchestra.run.smithi023.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 253 kB/s | 25 kB 00:00 2023-11-16T23:46:48.161 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:46:48.176 INFO:teuthology.orchestra.run.smithi023.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 985 kB/s | 49 kB 00:00 2023-11-16T23:46:48.209 INFO:teuthology.orchestra.run.smithi023.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 597 kB/s | 94 kB 00:00 2023-11-16T23:46:48.211 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T23:46:48.243 INFO:teuthology.orchestra.run.smithi023.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.6 MB/s | 524 kB 00:00 2023-11-16T23:46:48.268 INFO:teuthology.orchestra.run.smithi023.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-11-16T23:46:48.294 INFO:teuthology.orchestra.run.smithi023.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-11-16T23:46:48.339 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T23:46:48.427 INFO:teuthology.orchestra.run.smithi023.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 16 MB/s | 2.2 MB 00:00 2023-11-16T23:46:48.442 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:46:48.564 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T23:46:48.586 INFO:teuthology.orchestra.run.smithi023.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-11-16T23:46:48.636 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T23:46:48.679 INFO:teuthology.orchestra.run.smithi023.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 16 MB/s | 8.0 MB 00:00 2023-11-16T23:46:48.679 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:46:48.679 INFO:teuthology.orchestra.run.smithi023.stdout:Total 29 MB/s | 18 MB 00:00 2023-11-16T23:46:48.847 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T23:46:48.924 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T23:46:49.027 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-11-16T23:46:49.052 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-11-16T23:46:49.052 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-11-16T23:46:49.188 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T23:46:49.555 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-11-16T23:46:49.560 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-11-16T23:46:49.713 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T23:46:49.948 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T23:46:50.113 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-11-16T23:46:50.373 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T23:46:50.566 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:46:50.591 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T23:46:50.787 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T23:46:50.818 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T23:46:50.865 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T23:46:50.985 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T23:46:51.045 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T23:46:51.095 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T23:46:51.179 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:46:51.333 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T23:46:51.427 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T23:46:51.446 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:46:51.446 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T23:46:51.446 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:46:51.446 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T23:46:51.446 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T23:46:51.447 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T23:46:51.447 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T23:46:51.447 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T23:46:51.447 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T23:46:51.447 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T23:46:51.518 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:46:51.781 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T23:46:51.782 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T23:46:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T23:46:52.664 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T23:46:52.664 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:05:15 ago on Thu 16 Nov 2023 11:41:37 PM UTC. 2023-11-16T23:46:52.921 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T23:46:52.921 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:05:13 ago on Thu 16 Nov 2023 11:41:39 PM UTC. 2023-11-16T23:46:53.254 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T23:46:53.572 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T23:46:53.762 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T23:47:02.042 INFO:teuthology.orchestra.run.smithi173.stdout:Installed products updated. 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:02.043 INFO:teuthology.orchestra.run.smithi173.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T23:47:02.044 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:47:02.044 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T23:47:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:47:02.106 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-11-16T23:47:02.107 DEBUG:teuthology.parallel:result is None 2023-11-16T23:47:02.107 DEBUG:teuthology.parallel:result is None 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T23:47:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T23:47:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T23:47:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T23:47:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T23:47:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T23:47:03.246 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:08 ago on Thu 16 Nov 2023 11:44:47 PM UTC. 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T23:47:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-11-16T23:47:03.276 DEBUG:teuthology.parallel:result is None 2023-11-16T23:47:03.276 INFO:teuthology.run_tasks:Running task install... 2023-11-16T23:47:03.285 DEBUG:teuthology.task.install:project ceph 2023-11-16T23:47:03.285 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b'}} 2023-11-16T23:47:03.285 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b'} 2023-11-16T23:47:03.285 INFO:teuthology.task.install:Using flavor: default 2023-11-16T23:47:03.291 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T23:47:03.291 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-16T23:47:03.292 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b', 'tag': None, 'wait_for_package': False} 2023-11-16T23:47:03.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:47:03.294 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b', 'tag': None, 'wait_for_package': False} 2023-11-16T23:47:03.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:47:03.297 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b', 'tag': None, 'wait_for_package': False} 2023-11-16T23:47:03.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:47:03.444 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/ 2023-11-16T23:47:03.445 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3906.g1c956514 2023-11-16T23:47:03.450 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/ 2023-11-16T23:47:03.450 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3906.g1c956514 2023-11-16T23:47:03.526 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/ 2023-11-16T23:47:03.527 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3906.g1c956514 2023-11-16T23:47:03.562 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T23:47:03.562 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:47:03.562 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T23:47:03.595 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T23:47:03.596 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1c9565146c8647d4e00eb6c7ae626dc192680e7b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T23:47:03.611 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T23:47:03.611 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:47:03.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T23:47:03.632 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T23:47:03.632 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:47:03.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T23:47:03.660 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T23:47:03.660 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1c9565146c8647d4e00eb6c7ae626dc192680e7b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T23:47:03.663 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T23:47:03.695 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T23:47:03.695 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1c9565146c8647d4e00eb6c7ae626dc192680e7b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T23:47:03.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T23:47:03.739 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T23:47:03.805 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T23:47:03.806 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2023-11-16T23:47:03.807 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2023-11-16T23:47:03.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T23:47:03.871 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2023-11-16T23:47:03.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2023-11-16T23:47:03.906 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T23:47:03.976 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2023-11-16T23:47:03.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2023-11-16T23:47:04.189 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:04.267 INFO:teuthology.orchestra.run.smithi173.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:04.458 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:04.688 INFO:teuthology.orchestra.run.smithi156.stdout:44 files removed 2023-11-16T23:47:04.704 INFO:teuthology.orchestra.run.smithi173.stdout:44 files removed 2023-11-16T23:47:04.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T23:47:04.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T23:47:04.898 INFO:teuthology.orchestra.run.smithi023.stdout:44 files removed 2023-11-16T23:47:04.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T23:47:05.084 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:05.119 INFO:teuthology.orchestra.run.smithi173.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:05.442 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-11-16T23:47:05.882 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 219 kB/s | 80 kB 00:00 2023-11-16T23:47:05.906 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 220 kB/s | 80 kB 00:00 2023-11-16T23:47:06.174 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-11-16T23:47:06.184 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 35 kB/s | 9.4 kB 00:00 2023-11-16T23:47:06.230 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 252 kB/s | 80 kB 00:00 2023-11-16T23:47:06.443 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2023-11-16T23:47:06.545 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2023-11-16T23:47:06.586 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-16T23:47:06.631 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-11-16T23:47:06.746 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-11-16T23:47:07.015 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-11-16T23:47:07.158 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-11-16T23:47:11.352 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 16 MB 00:04 2023-11-16T23:47:12.418 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 16 MB 00:05 2023-11-16T23:47:14.566 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 2.0 MB/s | 16 MB 00:07 2023-11-16T23:47:14.892 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 410 kB/s | 24 kB 00:00 2023-11-16T23:47:15.819 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 MB/s | 47 MB 00:00 2023-11-16T23:47:16.758 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 488 kB/s | 24 kB 00:00 2023-11-16T23:47:17.934 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-11-16T23:47:18.118 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 277 kB/s | 24 kB 00:00 2023-11-16T23:47:19.051 INFO:teuthology.orchestra.run.smithi173.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-11-16T23:47:24.162 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 MB/s | 53 MB 00:01 2023-11-16T23:47:27.211 INFO:teuthology.orchestra.run.smithi173.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 MB/s | 47 MB 00:00 2023-11-16T23:47:27.253 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-11-16T23:47:35.394 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:35.395 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:35.411 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:35.411 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:35.413 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T23:47:35.414 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T23:47:35.416 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T23:47:35.420 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T23:47:35.421 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T23:47:35.552 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.4 k 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:18.0.0-3906.g1c956514.el8 ceph 5.3 M 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 848 k 2023-11-16T23:47:35.565 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3906.g1c956514.el8 ceph 165 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:18.0.0-3906.g1c956514.el8 ceph 1.5 M 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 127 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 1.7 M 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 7.4 M 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 47 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 13 M 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:18.0.0-3906.g1c956514.el8 ceph 41 M 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 265 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 83 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 258 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 18 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T23:47:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 31 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 771 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 139 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:18.0.0-3906.g1c956514.el8 ceph 197 k 2023-11-16T23:47:35.567 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:18.0.0-3906.g1c956514.el8 ceph 371 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 364 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 107 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 94 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.4 M 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 180 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.7 M 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.6 M 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-11-16T23:47:35.568 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 22 M 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 23 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:18.0.0-3906.g1c956514.el8 ceph 2.3 M 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 240 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:18.0.0-3906.g1c956514.el8 ceph 4.6 M 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 18 M 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 15 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:18.0.0-3906.g1c956514.el8 ceph 24 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T23:47:35.569 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:18.0.0-3906.g1c956514.el8 ceph 137 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 473 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.5 M 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T23:47:35.570 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T23:47:35.571 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 45 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 119 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T23:47:35.572 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T23:47:35.573 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T23:47:35.574 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T23:47:35.575 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql 10 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout:Install 128 Packages 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 3 Packages 2023-11-16T23:47:35.576 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:47:35.577 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 197 M 2023-11-16T23:47:35.577 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-11-16T23:47:35.829 INFO:teuthology.orchestra.run.smithi156.stdout:(1/131): ceph-18.0.0-3906.g1c956514.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-11-16T23:47:36.154 INFO:teuthology.orchestra.run.smithi156.stdout:(2/131): ceph-fuse-18.0.0-3906.g1c956514.el8.x8 2.5 MB/s | 848 kB 00:00 2023-11-16T23:47:36.248 INFO:teuthology.orchestra.run.smithi156.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 1.8 MB/s | 165 kB 00:00 2023-11-16T23:47:36.788 INFO:teuthology.orchestra.run.smithi156.stdout:(4/131): ceph-base-18.0.0-3906.g1c956514.el8.x8 4.6 MB/s | 5.3 MB 00:01 2023-11-16T23:47:36.847 INFO:teuthology.orchestra.run.smithi156.stdout:(5/131): ceph-mds-18.0.0-3906.g1c956514.el8.x86 3.8 MB/s | 2.3 MB 00:00 2023-11-16T23:47:37.072 INFO:teuthology.orchestra.run.smithi156.stdout:(6/131): ceph-mgr-18.0.0-3906.g1c956514.el8.x86 5.5 MB/s | 1.5 MB 00:00 2023-11-16T23:47:37.297 INFO:teuthology.orchestra.run.smithi156.stdout:(7/131): ceph-common-18.0.0-3906.g1c956514.el8. 13 MB/s | 22 MB 00:01 2023-11-16T23:47:37.789 INFO:teuthology.orchestra.run.smithi156.stdout:(8/131): ceph-mon-18.0.0-3906.g1c956514.el8.x86 4.9 MB/s | 4.6 MB 00:00 2023-11-16T23:47:37.889 INFO:teuthology.orchestra.run.smithi156.stdout:(9/131): ceph-selinux-18.0.0-3906.g1c956514.el8 244 kB/s | 24 kB 00:00 2023-11-16T23:47:38.301 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:38.302 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:38.316 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:38.316 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:38.318 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T23:47:38.319 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T23:47:38.321 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T23:47:38.324 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T23:47:38.325 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T23:47:38.381 INFO:teuthology.orchestra.run.smithi156.stdout:(10/131): ceph-radosgw-18.0.0-3906.g1c956514.el 12 MB/s | 13 MB 00:01 2023-11-16T23:47:38.457 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-11-16T23:47:38.469 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================================================== 2023-11-16T23:47:38.469 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-11-16T23:47:38.469 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================================================== 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.4 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:18.0.0-3906.g1c956514.el8 ceph 5.3 M 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 848 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3906.g1c956514.el8 ceph 165 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:18.0.0-3906.g1c956514.el8 ceph 1.5 M 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 127 k 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 1.7 M 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 7.4 M 2023-11-16T23:47:38.470 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 47 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 13 M 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:18.0.0-3906.g1c956514.el8 ceph 41 M 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 265 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 83 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 258 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 18 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 31 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 771 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 139 k 2023-11-16T23:47:38.471 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:18.0.0-3906.g1c956514.el8 ceph 197 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:18.0.0-3906.g1c956514.el8 ceph 371 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 364 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 107 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 94 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.4 M 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 180 k 2023-11-16T23:47:38.472 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.7 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.6 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 22 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 23 k 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:18.0.0-3906.g1c956514.el8 ceph 2.3 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 240 k 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:18.0.0-3906.g1c956514.el8 ceph 4.6 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 18 M 2023-11-16T23:47:38.473 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 15 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:18.0.0-3906.g1c956514.el8 ceph 24 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:18.0.0-3906.g1c956514.el8 ceph 137 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T23:47:38.474 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 473 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.5 M 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T23:47:38.475 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 45 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 119 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T23:47:38.476 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T23:47:38.477 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T23:47:38.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T23:47:38.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T23:47:38.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T23:47:38.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T23:47:38.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T23:47:38.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T23:47:38.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T23:47:38.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T23:47:38.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T23:47:38.480 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T23:47:38.481 INFO:teuthology.orchestra.run.smithi173.stdout:Enabling module streams: 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql 10 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================================================== 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout:Install 128 Packages 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 3 Packages 2023-11-16T23:47:38.482 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:47:38.483 INFO:teuthology.orchestra.run.smithi156.stdout:(11/131): cephfs-mirror-18.0.0-3906.g1c956514.e 2.5 MB/s | 258 kB 00:00 2023-11-16T23:47:38.483 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 197 M 2023-11-16T23:47:38.483 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-11-16T23:47:38.557 INFO:teuthology.orchestra.run.smithi156.stdout:(12/131): libcephfs-devel-18.0.0-3906.g1c956514 413 kB/s | 31 kB 00:00 2023-11-16T23:47:38.707 INFO:teuthology.orchestra.run.smithi156.stdout:(13/131): libcephfs2-18.0.0-3906.g1c956514.el8. 5.0 MB/s | 771 kB 00:00 2023-11-16T23:47:38.791 INFO:teuthology.orchestra.run.smithi156.stdout:(14/131): libcephsqlite-18.0.0-3906.g1c956514.e 1.6 MB/s | 137 kB 00:00 2023-11-16T23:47:38.858 INFO:teuthology.orchestra.run.smithi156.stdout:(15/131): librados-devel-18.0.0-3906.g1c956514. 2.0 MB/s | 139 kB 00:00 2023-11-16T23:47:38.989 INFO:teuthology.orchestra.run.smithi173.stdout:(1/131): ceph-18.0.0-3906.g1c956514.el8.x86_64. 44 kB/s | 6.4 kB 00:00 2023-11-16T23:47:39.058 INFO:teuthology.orchestra.run.smithi156.stdout:(16/131): ceph-osd-18.0.0-3906.g1c956514.el8.x8 8.9 MB/s | 18 MB 00:01 2023-11-16T23:47:39.084 INFO:teuthology.orchestra.run.smithi156.stdout:(17/131): libradosstriper1-18.0.0-3906.g1c95651 2.1 MB/s | 473 kB 00:00 2023-11-16T23:47:39.142 INFO:teuthology.orchestra.run.smithi156.stdout:(18/131): python3-ceph-argparse-18.0.0-3906.g1c 765 kB/s | 45 kB 00:00 2023-11-16T23:47:39.243 INFO:teuthology.orchestra.run.smithi156.stdout:(19/131): python3-ceph-common-18.0.0-3906.g1c95 1.2 MB/s | 119 kB 00:00 2023-11-16T23:47:39.334 INFO:teuthology.orchestra.run.smithi156.stdout:(20/131): python3-cephfs-18.0.0-3906.g1c956514. 2.1 MB/s | 197 kB 00:00 2023-11-16T23:47:39.443 INFO:teuthology.orchestra.run.smithi156.stdout:(21/131): python3-rados-18.0.0-3906.g1c956514.e 3.3 MB/s | 371 kB 00:00 2023-11-16T23:47:39.487 INFO:teuthology.orchestra.run.smithi173.stdout:(2/131): ceph-fuse-18.0.0-3906.g1c956514.el8.x8 1.7 MB/s | 848 kB 00:00 2023-11-16T23:47:39.527 INFO:teuthology.orchestra.run.smithi156.stdout:(22/131): python3-rbd-18.0.0-3906.g1c956514.el8 4.3 MB/s | 364 kB 00:00 2023-11-16T23:47:39.610 INFO:teuthology.orchestra.run.smithi156.stdout:(23/131): python3-rgw-18.0.0-3906.g1c956514.el8 1.3 MB/s | 107 kB 00:00 2023-11-16T23:47:39.613 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:39.614 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T23:47:39.630 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:39.631 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T23:47:39.633 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T23:47:39.634 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T23:47:39.636 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T23:47:39.639 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T23:47:39.641 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T23:47:39.754 INFO:teuthology.orchestra.run.smithi173.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 621 kB/s | 165 kB 00:00 2023-11-16T23:47:39.791 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-11-16T23:47:39.805 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-11-16T23:47:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-11-16T23:47:39.805 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-11-16T23:47:39.805 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.4 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:18.0.0-3906.g1c956514.el8 ceph 5.3 M 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 848 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3906.g1c956514.el8 ceph 165 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:18.0.0-3906.g1c956514.el8 ceph 1.5 M 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 127 k 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 1.7 M 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 7.4 M 2023-11-16T23:47:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 47 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 13 M 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:18.0.0-3906.g1c956514.el8 ceph 41 M 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 265 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 83 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 258 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 18 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 31 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 771 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:18.0.0-3906.g1c956514.el8 ceph 139 k 2023-11-16T23:47:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:18.0.0-3906.g1c956514.el8 ceph 197 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:18.0.0-3906.g1c956514.el8 ceph 371 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 364 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:18.0.0-3906.g1c956514.el8 ceph 107 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 94 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.4 M 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 180 k 2023-11-16T23:47:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.7 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 3.6 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 22 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 23 k 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:18.0.0-3906.g1c956514.el8 ceph 2.3 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 240 k 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:18.0.0-3906.g1c956514.el8 ceph 4.6 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:18.0.0-3906.g1c956514.el8 ceph 18 M 2023-11-16T23:47:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3906.g1c956514.el8 ceph-noarch 15 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:18.0.0-3906.g1c956514.el8 ceph 24 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:18.0.0-3906.g1c956514.el8 ceph 137 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T23:47:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 473 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:18.0.0-3906.g1c956514.el8 ceph 6.5 M 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T23:47:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T23:47:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:18.0.0-3906.g1c956514.el8 ceph 45 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:18.0.0-3906.g1c956514.el8 ceph 119 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T23:47:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T23:47:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T23:47:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T23:47:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql 10 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout:Install 128 Packages 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 3 Packages 2023-11-16T23:47:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:47:39.818 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 197 M 2023-11-16T23:47:39.818 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-11-16T23:47:39.878 INFO:teuthology.orchestra.run.smithi156.stdout:(24/131): rbd-fuse-18.0.0-3906.g1c956514.el8.x8 352 kB/s | 94 kB 00:00 2023-11-16T23:47:40.036 INFO:teuthology.orchestra.run.smithi156.stdout:(25/131): librgw2-18.0.0-3906.g1c956514.el8.x86 6.7 MB/s | 6.5 MB 00:00 2023-11-16T23:47:40.103 INFO:teuthology.orchestra.run.smithi023.stdout:(1/131): ceph-18.0.0-3906.g1c956514.el8.x86_64. 26 kB/s | 6.4 kB 00:00 2023-11-16T23:47:40.145 INFO:teuthology.orchestra.run.smithi156.stdout:(26/131): rbd-nbd-18.0.0-3906.g1c956514.el8.x86 1.6 MB/s | 180 kB 00:00 2023-11-16T23:47:40.287 INFO:teuthology.orchestra.run.smithi156.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3906.g 163 kB/s | 23 kB 00:00 2023-11-16T23:47:40.395 INFO:teuthology.orchestra.run.smithi156.stdout:(28/131): rbd-mirror-18.0.0-3906.g1c956514.el8. 6.7 MB/s | 3.4 MB 00:00 2023-11-16T23:47:40.421 INFO:teuthology.orchestra.run.smithi156.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3906.g1c95651 953 kB/s | 127 kB 00:00 2023-11-16T23:47:40.496 INFO:teuthology.orchestra.run.smithi173.stdout:(4/131): ceph-base-18.0.0-3906.g1c956514.el8.x8 3.2 MB/s | 5.3 MB 00:01 2023-11-16T23:47:40.738 INFO:teuthology.orchestra.run.smithi156.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3906.g1c956 5.0 MB/s | 1.7 MB 00:00 2023-11-16T23:47:40.854 INFO:teuthology.orchestra.run.smithi156.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3906.g1c 2.0 MB/s | 240 kB 00:00 2023-11-16T23:47:40.879 INFO:teuthology.orchestra.run.smithi173.stdout:(5/131): ceph-mgr-18.0.0-3906.g1c956514.el8.x86 4.0 MB/s | 1.5 MB 00:00 2023-11-16T23:47:40.913 INFO:teuthology.orchestra.run.smithi156.stdout:(32/131): ceph-mgr-rook-18.0.0-3906.g1c956514.e 814 kB/s | 47 kB 00:00 2023-11-16T23:47:40.970 INFO:teuthology.orchestra.run.smithi023.stdout:(2/131): ceph-fuse-18.0.0-3906.g1c956514.el8.x8 978 kB/s | 848 kB 00:00 2023-11-16T23:47:41.022 INFO:teuthology.orchestra.run.smithi156.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3906.g1 134 kB/s | 15 kB 00:00 2023-11-16T23:47:41.137 INFO:teuthology.orchestra.run.smithi023.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 993 kB/s | 165 kB 00:00 2023-11-16T23:47:41.164 INFO:teuthology.orchestra.run.smithi156.stdout:(34/131): ceph-volume-18.0.0-3906.g1c956514.el8 1.8 MB/s | 265 kB 00:00 2023-11-16T23:47:41.247 INFO:teuthology.orchestra.run.smithi156.stdout:(35/131): cephadm-18.0.0-3906.g1c956514.el8.noa 1.0 MB/s | 83 kB 00:00 2023-11-16T23:47:41.323 INFO:teuthology.orchestra.run.smithi156.stdout:(36/131): cephfs-top-18.0.0-3906.g1c956514.el8. 239 kB/s | 18 kB 00:00 2023-11-16T23:47:41.429 INFO:teuthology.orchestra.run.smithi173.stdout:(6/131): ceph-mds-18.0.0-3906.g1c956514.el8.x86 1.4 MB/s | 2.3 MB 00:01 2023-11-16T23:47:41.723 INFO:teuthology.orchestra.run.smithi156.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 5.7 MB/s | 7.4 MB 00:01 2023-11-16T23:47:41.771 INFO:teuthology.orchestra.run.smithi173.stdout:(7/131): ceph-mon-18.0.0-3906.g1c956514.el8.x86 5.2 MB/s | 4.6 MB 00:00 2023-11-16T23:47:42.237 INFO:teuthology.orchestra.run.smithi023.stdout:(4/131): ceph-mds-18.0.0-3906.g1c956514.el8.x86 2.1 MB/s | 2.3 MB 00:01 2023-11-16T23:47:42.298 INFO:teuthology.orchestra.run.smithi156.stdout:(38/131): ceph-test-18.0.0-3906.g1c956514.el8.x 9.2 MB/s | 41 MB 00:04 2023-11-16T23:47:42.754 INFO:teuthology.orchestra.run.smithi023.stdout:(5/131): ceph-mgr-18.0.0-3906.g1c956514.el8.x86 3.0 MB/s | 1.5 MB 00:00 2023-11-16T23:47:42.882 INFO:teuthology.orchestra.run.smithi156.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 196 kB/s | 306 kB 00:01 2023-11-16T23:47:43.182 INFO:teuthology.orchestra.run.smithi156.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 40 kB/s | 59 kB 00:01 2023-11-16T23:47:43.330 INFO:teuthology.orchestra.run.smithi173.stdout:(8/131): ceph-radosgw-18.0.0-3906.g1c956514.el8 8.4 MB/s | 13 MB 00:01 2023-11-16T23:47:43.388 INFO:teuthology.orchestra.run.smithi173.stdout:(9/131): ceph-selinux-18.0.0-3906.g1c956514.el8 419 kB/s | 24 kB 00:00 2023-11-16T23:47:43.408 INFO:teuthology.orchestra.run.smithi156.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 906 kB/s | 476 kB 00:00 2023-11-16T23:47:43.458 INFO:teuthology.orchestra.run.smithi156.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 65 kB/s | 75 kB 00:01 2023-11-16T23:47:43.492 INFO:teuthology.orchestra.run.smithi156.stdout:(43/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 141 kB/s | 44 kB 00:00 2023-11-16T23:47:43.675 INFO:teuthology.orchestra.run.smithi156.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 693 kB/s | 185 kB 00:00 2023-11-16T23:47:43.712 INFO:teuthology.orchestra.run.smithi023.stdout:(6/131): ceph-base-18.0.0-3906.g1c956514.el8.x8 1.4 MB/s | 5.3 MB 00:03 2023-11-16T23:47:43.722 INFO:teuthology.orchestra.run.smithi173.stdout:(10/131): ceph-common-18.0.0-3906.g1c956514.el8 4.4 MB/s | 22 MB 00:04 2023-11-16T23:47:43.751 INFO:teuthology.orchestra.run.smithi156.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 63 kB/s | 16 kB 00:00 2023-11-16T23:47:43.801 INFO:teuthology.orchestra.run.smithi156.stdout:(46/131): python3-cachetools-3.1.1-4.el8.noarch 96 kB/s | 33 kB 00:00 2023-11-16T23:47:43.814 INFO:teuthology.orchestra.run.smithi173.stdout:(11/131): cephfs-mirror-18.0.0-3906.g1c956514.e 2.8 MB/s | 258 kB 00:00 2023-11-16T23:47:43.875 INFO:teuthology.orchestra.run.smithi173.stdout:(12/131): libcephfs-devel-18.0.0-3906.g1c956514 534 kB/s | 31 kB 00:00 2023-11-16T23:47:43.951 INFO:teuthology.orchestra.run.smithi156.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 626 kB/s | 173 kB 00:00 2023-11-16T23:47:43.996 INFO:teuthology.orchestra.run.smithi023.stdout:(7/131): ceph-mon-18.0.0-3906.g1c956514.el8.x86 3.7 MB/s | 4.6 MB 00:01 2023-11-16T23:47:44.006 INFO:teuthology.orchestra.run.smithi173.stdout:(13/131): libcephfs2-18.0.0-3906.g1c956514.el8. 5.6 MB/s | 771 kB 00:00 2023-11-16T23:47:44.065 INFO:teuthology.orchestra.run.smithi173.stdout:(14/131): libcephsqlite-18.0.0-3906.g1c956514.e 2.3 MB/s | 137 kB 00:00 2023-11-16T23:47:44.149 INFO:teuthology.orchestra.run.smithi173.stdout:(15/131): librados-devel-18.0.0-3906.g1c956514. 1.6 MB/s | 139 kB 00:00 2023-11-16T23:47:44.185 INFO:teuthology.orchestra.run.smithi156.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 48 kB/s | 11 kB 00:00 2023-11-16T23:47:44.210 INFO:teuthology.orchestra.run.smithi156.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 199 kB/s | 81 kB 00:00 2023-11-16T23:47:44.282 INFO:teuthology.orchestra.run.smithi173.stdout:(16/131): libradosstriper1-18.0.0-3906.g1c95651 3.5 MB/s | 473 kB 00:00 2023-11-16T23:47:44.410 INFO:teuthology.orchestra.run.smithi156.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 78 kB/s | 18 kB 00:00 2023-11-16T23:47:44.511 INFO:teuthology.orchestra.run.smithi156.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 506 kB/s | 384 kB 00:00 2023-11-16T23:47:44.652 INFO:teuthology.orchestra.run.smithi156.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 201 kB/s | 49 kB 00:00 2023-11-16T23:47:44.753 INFO:teuthology.orchestra.run.smithi156.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 243 kB/s | 59 kB 00:00 2023-11-16T23:47:45.028 INFO:teuthology.orchestra.run.smithi156.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 160 kB/s | 60 kB 00:00 2023-11-16T23:47:45.070 INFO:teuthology.orchestra.run.smithi156.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 891 kB/s | 283 kB 00:00 2023-11-16T23:47:45.262 INFO:teuthology.orchestra.run.smithi156.stdout:(56/131): python3-kubernetes-11.0.0-6.el8.noarc 1.1 MB/s | 1.2 MB 00:01 2023-11-16T23:47:45.279 INFO:teuthology.orchestra.run.smithi156.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 62 kB/s | 16 kB 00:00 2023-11-16T23:47:45.440 INFO:teuthology.orchestra.run.smithi173.stdout:(17/131): librgw2-18.0.0-3906.g1c956514.el8.x86 5.6 MB/s | 6.5 MB 00:01 2023-11-16T23:47:45.491 INFO:teuthology.orchestra.run.smithi173.stdout:(18/131): python3-ceph-argparse-18.0.0-3906.g1c 899 kB/s | 45 kB 00:00 2023-11-16T23:47:45.512 INFO:teuthology.orchestra.run.smithi156.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 76 kB/s | 34 kB 00:00 2023-11-16T23:47:45.538 INFO:teuthology.orchestra.run.smithi156.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 248 kB/s | 64 kB 00:00 2023-11-16T23:47:45.558 INFO:teuthology.orchestra.run.smithi173.stdout:(19/131): python3-ceph-common-18.0.0-3906.g1c95 1.7 MB/s | 119 kB 00:00 2023-11-16T23:47:45.563 INFO:teuthology.orchestra.run.smithi156.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 653 kB/s | 196 kB 00:00 2023-11-16T23:47:45.624 INFO:teuthology.orchestra.run.smithi173.stdout:(20/131): python3-cephfs-18.0.0-3906.g1c956514. 2.9 MB/s | 197 kB 00:00 2023-11-16T23:47:45.725 INFO:teuthology.orchestra.run.smithi173.stdout:(21/131): python3-rados-18.0.0-3906.g1c956514.e 3.6 MB/s | 371 kB 00:00 2023-11-16T23:47:45.755 INFO:teuthology.orchestra.run.smithi156.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 77 kB/s | 19 kB 00:00 2023-11-16T23:47:45.780 INFO:teuthology.orchestra.run.smithi156.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 99 kB/s | 24 kB 00:00 2023-11-16T23:47:45.806 INFO:teuthology.orchestra.run.smithi156.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 121 kB/s | 29 kB 00:00 2023-11-16T23:47:45.825 INFO:teuthology.orchestra.run.smithi173.stdout:(22/131): python3-rbd-18.0.0-3906.g1c956514.el8 3.6 MB/s | 364 kB 00:00 2023-11-16T23:47:45.892 INFO:teuthology.orchestra.run.smithi173.stdout:(23/131): python3-rgw-18.0.0-3906.g1c956514.el8 1.6 MB/s | 107 kB 00:00 2023-11-16T23:47:45.998 INFO:teuthology.orchestra.run.smithi156.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 114 kB/s | 27 kB 00:00 2023-11-16T23:47:46.039 INFO:teuthology.orchestra.run.smithi156.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 986 kB/s | 255 kB 00:00 2023-11-16T23:47:46.067 INFO:teuthology.orchestra.run.smithi173.stdout:(24/131): ceph-osd-18.0.0-3906.g1c956514.el8.x8 3.8 MB/s | 18 MB 00:04 2023-11-16T23:47:46.092 INFO:teuthology.orchestra.run.smithi173.stdout:(25/131): rbd-fuse-18.0.0-3906.g1c956514.el8.x8 472 kB/s | 94 kB 00:00 2023-11-16T23:47:46.112 INFO:teuthology.orchestra.run.smithi156.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 819 kB/s | 251 kB 00:00 2023-11-16T23:47:46.151 INFO:teuthology.orchestra.run.smithi173.stdout:(26/131): rbd-nbd-18.0.0-3906.g1c956514.el8.x86 3.0 MB/s | 180 kB 00:00 2023-11-16T23:47:46.226 INFO:teuthology.orchestra.run.smithi173.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3906.g 308 kB/s | 23 kB 00:00 2023-11-16T23:47:46.257 INFO:teuthology.orchestra.run.smithi156.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 236 kB/s | 61 kB 00:00 2023-11-16T23:47:46.282 INFO:teuthology.orchestra.run.smithi156.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 352 kB/s | 85 kB 00:00 2023-11-16T23:47:46.285 INFO:teuthology.orchestra.run.smithi173.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3906.g1c95651 2.1 MB/s | 127 kB 00:00 2023-11-16T23:47:46.338 INFO:teuthology.orchestra.run.smithi023.stdout:(8/131): ceph-common-18.0.0-3906.g1c956514.el8. 3.3 MB/s | 22 MB 00:06 2023-11-16T23:47:46.341 INFO:teuthology.orchestra.run.smithi156.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 70 kB 00:00 2023-11-16T23:47:46.368 INFO:teuthology.orchestra.run.smithi156.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 90 kB/s | 23 kB 00:00 2023-11-16T23:47:46.396 INFO:teuthology.orchestra.run.smithi023.stdout:(9/131): ceph-selinux-18.0.0-3906.g1c956514.el8 419 kB/s | 24 kB 00:00 2023-11-16T23:47:46.442 INFO:teuthology.orchestra.run.smithi156.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-11-16T23:47:46.525 INFO:teuthology.orchestra.run.smithi156.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 650 kB/s | 54 kB 00:00 2023-11-16T23:47:46.584 INFO:teuthology.orchestra.run.smithi156.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.4 MB/s | 259 kB 00:00 2023-11-16T23:47:46.642 INFO:teuthology.orchestra.run.smithi156.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-11-16T23:47:46.693 INFO:teuthology.orchestra.run.smithi156.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 778 kB/s | 39 kB 00:00 2023-11-16T23:47:46.751 INFO:teuthology.orchestra.run.smithi156.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2023-11-16T23:47:46.795 INFO:teuthology.orchestra.run.smithi173.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3906.g1c956 3.4 MB/s | 1.7 MB 00:00 2023-11-16T23:47:46.897 INFO:teuthology.orchestra.run.smithi023.stdout:(10/131): ceph-radosgw-18.0.0-3906.g1c956514.el 4.5 MB/s | 13 MB 00:02 2023-11-16T23:47:46.927 INFO:teuthology.orchestra.run.smithi156.stdout:(77/131): python3-scipy-1.0.0-21.module+el8.5.0 23 MB/s | 14 MB 00:00 2023-11-16T23:47:46.961 INFO:teuthology.orchestra.run.smithi173.stdout:(30/131): rbd-mirror-18.0.0-3906.g1c956514.el8. 3.9 MB/s | 3.4 MB 00:00 2023-11-16T23:47:46.968 INFO:teuthology.orchestra.run.smithi156.stdout:(78/131): thrift-0.13.0-2.el8.x86_64.rpm 2.4 MB/s | 1.7 MB 00:00 2023-11-16T23:47:46.971 INFO:teuthology.orchestra.run.smithi023.stdout:(11/131): cephfs-mirror-18.0.0-3906.g1c956514.e 3.4 MB/s | 258 kB 00:00 2023-11-16T23:47:46.994 INFO:teuthology.orchestra.run.smithi156.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-11-16T23:47:47.022 INFO:teuthology.orchestra.run.smithi023.stdout:(12/131): libcephfs-devel-18.0.0-3906.g1c956514 620 kB/s | 31 kB 00:00 2023-11-16T23:47:47.027 INFO:teuthology.orchestra.run.smithi156.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.9 MB/s | 538 kB 00:00 2023-11-16T23:47:47.053 INFO:teuthology.orchestra.run.smithi156.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2023-11-16T23:47:47.079 INFO:teuthology.orchestra.run.smithi173.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3906.g1c 2.0 MB/s | 240 kB 00:00 2023-11-16T23:47:47.086 INFO:teuthology.orchestra.run.smithi156.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 8.0 MB/s | 713 kB 00:00 2023-11-16T23:47:47.103 INFO:teuthology.orchestra.run.smithi156.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 528 kB/s | 27 kB 00:00 2023-11-16T23:47:47.137 INFO:teuthology.orchestra.run.smithi156.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2023-11-16T23:47:47.144 INFO:teuthology.orchestra.run.smithi173.stdout:(32/131): ceph-mgr-rook-18.0.0-3906.g1c956514.e 727 kB/s | 47 kB 00:00 2023-11-16T23:47:47.162 INFO:teuthology.orchestra.run.smithi156.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 747 kB/s | 44 kB 00:00 2023-11-16T23:47:47.197 INFO:teuthology.orchestra.run.smithi023.stdout:(13/131): libcephfs2-18.0.0-3906.g1c956514.el8. 4.3 MB/s | 771 kB 00:00 2023-11-16T23:47:47.203 INFO:teuthology.orchestra.run.smithi173.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3906.g1 249 kB/s | 15 kB 00:00 2023-11-16T23:47:47.221 INFO:teuthology.orchestra.run.smithi156.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2023-11-16T23:47:47.248 INFO:teuthology.orchestra.run.smithi023.stdout:(14/131): libcephsqlite-18.0.0-3906.g1c956514.e 2.7 MB/s | 137 kB 00:00 2023-11-16T23:47:47.312 INFO:teuthology.orchestra.run.smithi173.stdout:(34/131): ceph-volume-18.0.0-3906.g1c956514.el8 2.4 MB/s | 265 kB 00:00 2023-11-16T23:47:47.330 INFO:teuthology.orchestra.run.smithi156.stdout:(87/131): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-11-16T23:47:47.355 INFO:teuthology.orchestra.run.smithi156.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.6 MB/s | 225 kB 00:00 2023-11-16T23:47:47.370 INFO:teuthology.orchestra.run.smithi173.stdout:(35/131): cephadm-18.0.0-3906.g1c956514.el8.noa 1.4 MB/s | 83 kB 00:00 2023-11-16T23:47:47.380 INFO:teuthology.orchestra.run.smithi156.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-16T23:47:47.406 INFO:teuthology.orchestra.run.smithi156.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-11-16T23:47:47.420 INFO:teuthology.orchestra.run.smithi023.stdout:(15/131): librados-devel-18.0.0-3906.g1c956514. 806 kB/s | 139 kB 00:00 2023-11-16T23:47:47.429 INFO:teuthology.orchestra.run.smithi173.stdout:(36/131): cephfs-top-18.0.0-3906.g1c956514.el8. 309 kB/s | 18 kB 00:00 2023-11-16T23:47:47.430 INFO:teuthology.orchestra.run.smithi156.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 399 kB/s | 20 kB 00:00 2023-11-16T23:47:47.456 INFO:teuthology.orchestra.run.smithi156.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-11-16T23:47:47.514 INFO:teuthology.orchestra.run.smithi156.stdout:(93/131): python3-numpy-1.14.3-10.el8.x86_64.rp 9.9 MB/s | 3.7 MB 00:00 2023-11-16T23:47:47.548 INFO:teuthology.orchestra.run.smithi156.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 5.9 MB/s | 709 kB 00:00 2023-11-16T23:47:47.575 INFO:teuthology.orchestra.run.smithi156.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 3.2 MB/s | 198 kB 00:00 2023-11-16T23:47:47.626 INFO:teuthology.orchestra.run.smithi156.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2023-11-16T23:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout:(16/131): libradosstriper1-18.0.0-3906.g1c95651 1.7 MB/s | 473 kB 00:00 2023-11-16T23:47:47.734 INFO:teuthology.orchestra.run.smithi156.stdout:(97/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-11-16T23:47:47.768 INFO:teuthology.orchestra.run.smithi156.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-11-16T23:47:47.843 INFO:teuthology.orchestra.run.smithi156.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-11-16T23:47:47.885 INFO:teuthology.orchestra.run.smithi156.stdout:(100/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-11-16T23:47:47.927 INFO:teuthology.orchestra.run.smithi156.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-11-16T23:47:47.940 INFO:teuthology.orchestra.run.smithi173.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 6.5 MB/s | 7.4 MB 00:01 2023-11-16T23:47:47.969 INFO:teuthology.orchestra.run.smithi156.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-11-16T23:47:47.996 INFO:teuthology.orchestra.run.smithi156.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-11-16T23:47:48.020 INFO:teuthology.orchestra.run.smithi156.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 849 kB/s | 43 kB 00:00 2023-11-16T23:47:48.068 INFO:teuthology.orchestra.run.smithi156.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-11-16T23:47:48.104 INFO:teuthology.orchestra.run.smithi156.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 574 kB/s | 48 kB 00:00 2023-11-16T23:47:48.185 INFO:teuthology.orchestra.run.smithi173.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 442 kB/s | 306 kB 00:00 2023-11-16T23:47:48.185 INFO:teuthology.orchestra.run.smithi156.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.4 MB/s | 87 kB 00:00 2023-11-16T23:47:48.185 INFO:teuthology.orchestra.run.smithi156.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 848 kB/s | 43 kB 00:00 2023-11-16T23:47:48.186 INFO:teuthology.orchestra.run.smithi156.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-11-16T23:47:48.205 INFO:teuthology.orchestra.run.smithi156.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-11-16T23:47:48.230 INFO:teuthology.orchestra.run.smithi173.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 697 kB/s | 75 kB 00:00 2023-11-16T23:47:48.247 INFO:teuthology.orchestra.run.smithi156.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-11-16T23:47:48.272 INFO:teuthology.orchestra.run.smithi156.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 279 kB/s | 19 kB 00:00 2023-11-16T23:47:48.297 INFO:teuthology.orchestra.run.smithi156.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-11-16T23:47:48.331 INFO:teuthology.orchestra.run.smithi156.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-11-16T23:47:48.348 INFO:teuthology.orchestra.run.smithi156.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 355 kB/s | 18 kB 00:00 2023-11-16T23:47:48.415 INFO:teuthology.orchestra.run.smithi156.stdout:(116/131): postgresql-server-10.21-2.module+el8 7.4 MB/s | 5.1 MB 00:00 2023-11-16T23:47:48.466 INFO:teuthology.orchestra.run.smithi156.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2023-11-16T23:47:48.499 INFO:teuthology.orchestra.run.smithi156.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-11-16T23:47:48.524 INFO:teuthology.orchestra.run.smithi156.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-11-16T23:47:48.549 INFO:teuthology.orchestra.run.smithi156.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-11-16T23:47:48.575 INFO:teuthology.orchestra.run.smithi156.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-11-16T23:47:48.601 INFO:teuthology.orchestra.run.smithi156.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-11-16T23:47:48.622 INFO:teuthology.orchestra.run.smithi173.stdout:(40/131): ceph-test-18.0.0-3906.g1c956514.el8.x 7.8 MB/s | 41 MB 00:05 2023-11-16T23:47:48.624 INFO:teuthology.orchestra.run.smithi023.stdout:(17/131): librgw2-18.0.0-3906.g1c956514.el8.x86 7.0 MB/s | 6.5 MB 00:00 2023-11-16T23:47:48.626 INFO:teuthology.orchestra.run.smithi156.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-11-16T23:47:48.704 INFO:teuthology.orchestra.run.smithi023.stdout:(18/131): python3-ceph-argparse-18.0.0-3906.g1c 901 kB/s | 45 kB 00:00 2023-11-16T23:47:48.704 INFO:teuthology.orchestra.run.smithi173.stdout:(41/131): liboath-2.6.2-3.el8.x86_64.rpm 83 kB/s | 59 kB 00:00 2023-11-16T23:47:48.704 INFO:teuthology.orchestra.run.smithi173.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.0 MB/s | 476 kB 00:00 2023-11-16T23:47:48.705 INFO:teuthology.orchestra.run.smithi156.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 769 kB/s | 39 kB 00:00 2023-11-16T23:47:48.705 INFO:teuthology.orchestra.run.smithi156.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-11-16T23:47:48.710 INFO:teuthology.orchestra.run.smithi156.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-11-16T23:47:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:(19/131): python3-ceph-common-18.0.0-3906.g1c95 2.3 MB/s | 119 kB 00:00 2023-11-16T23:47:48.851 INFO:teuthology.orchestra.run.smithi023.stdout:(20/131): python3-cephfs-18.0.0-3906.g1c956514. 3.3 MB/s | 197 kB 00:00 2023-11-16T23:47:49.089 INFO:teuthology.orchestra.run.smithi156.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-11-16T23:47:49.089 INFO:teuthology.orchestra.run.smithi156.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.4 MB/s | 2.3 MB 00:00 2023-11-16T23:47:49.090 INFO:teuthology.orchestra.run.smithi156.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 10 MB/s | 798 kB 00:00 2023-11-16T23:47:49.090 INFO:teuthology.orchestra.run.smithi023.stdout:(21/131): python3-rados-18.0.0-3906.g1c956514.e 4.4 MB/s | 371 kB 00:00 2023-11-16T23:47:49.090 INFO:teuthology.orchestra.run.smithi023.stdout:(22/131): python3-rbd-18.0.0-3906.g1c956514.el8 4.3 MB/s | 364 kB 00:00 2023-11-16T23:47:49.091 INFO:teuthology.orchestra.run.smithi023.stdout:(23/131): python3-rgw-18.0.0-3906.g1c956514.el8 1.6 MB/s | 107 kB 00:00 2023-11-16T23:47:49.091 INFO:teuthology.orchestra.run.smithi173.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 264 kB/s | 33 kB 00:00 2023-11-16T23:47:49.091 INFO:teuthology.orchestra.run.smithi173.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 140 kB/s | 16 kB 00:00 2023-11-16T23:47:49.091 INFO:teuthology.orchestra.run.smithi173.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 453 kB/s | 185 kB 00:00 2023-11-16T23:47:49.091 INFO:teuthology.orchestra.run.smithi173.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-11-16T23:47:49.094 INFO:teuthology.orchestra.run.smithi023.stdout:(24/131): rbd-fuse-18.0.0-3906.g1c956514.el8.x8 1.2 MB/s | 94 kB 00:00 2023-11-16T23:47:49.126 INFO:teuthology.orchestra.run.smithi173.stdout:(47/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 87 kB/s | 44 kB 00:00 2023-11-16T23:47:49.217 INFO:teuthology.orchestra.run.smithi173.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 608 kB/s | 81 kB 00:00 2023-11-16T23:47:49.243 INFO:teuthology.orchestra.run.smithi173.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-11-16T23:47:49.326 INFO:teuthology.orchestra.run.smithi173.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2023-11-16T23:47:49.344 INFO:teuthology.orchestra.run.smithi156.stdout:(130/131): librados2-18.0.0-3906.g1c956514.el8. 5.8 MB/s | 3.7 MB 00:00 2023-11-16T23:47:49.411 INFO:teuthology.orchestra.run.smithi156.stdout:(131/131): librbd1-18.0.0-3906.g1c956514.el8.x8 5.3 MB/s | 3.6 MB 00:00 2023-11-16T23:47:49.416 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:47:49.416 INFO:teuthology.orchestra.run.smithi156.stdout:Total 14 MB/s | 197 MB 00:13 2023-11-16T23:47:49.435 INFO:teuthology.orchestra.run.smithi173.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 449 kB/s | 49 kB 00:00 2023-11-16T23:47:49.560 INFO:teuthology.orchestra.run.smithi173.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 470 kB/s | 59 kB 00:00 2023-11-16T23:47:49.636 INFO:teuthology.orchestra.run.smithi173.stdout:(53/131): python3-cherrypy-18.4.0-1.el8.noarch. 667 kB/s | 384 kB 00:00 2023-11-16T23:47:49.652 INFO:teuthology.orchestra.run.smithi173.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 656 kB/s | 60 kB 00:00 2023-11-16T23:47:49.734 INFO:teuthology.orchestra.run.smithi023.stdout:(25/131): ceph-osd-18.0.0-3906.g1c956514.el8.x8 2.9 MB/s | 18 MB 00:06 2023-11-16T23:47:49.745 INFO:teuthology.orchestra.run.smithi173.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 2.3 MB/s | 1.2 MB 00:00 2023-11-16T23:47:49.761 INFO:teuthology.orchestra.run.smithi173.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-11-16T23:47:49.793 INFO:teuthology.orchestra.run.smithi023.stdout:(26/131): rbd-mirror-18.0.0-3906.g1c956514.el8. 4.9 MB/s | 3.4 MB 00:00 2023-11-16T23:47:49.835 INFO:teuthology.orchestra.run.smithi023.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3906.g 556 kB/s | 23 kB 00:00 2023-11-16T23:47:49.837 INFO:teuthology.orchestra.run.smithi173.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 366 kB/s | 34 kB 00:00 2023-11-16T23:47:49.860 INFO:teuthology.orchestra.run.smithi023.stdout:(28/131): rbd-nbd-18.0.0-3906.g1c956514.el8.x86 1.4 MB/s | 180 kB 00:00 2023-11-16T23:47:49.862 INFO:teuthology.orchestra.run.smithi173.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-11-16T23:47:49.910 INFO:teuthology.orchestra.run.smithi023.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3906.g1c95651 1.7 MB/s | 127 kB 00:00 2023-11-16T23:47:49.977 INFO:teuthology.orchestra.run.smithi173.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 641 kB/s | 64 kB 00:00 2023-11-16T23:47:49.977 INFO:teuthology.orchestra.run.smithi173.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2023-11-16T23:47:49.981 INFO:teuthology.orchestra.run.smithi173.stdout:(61/131): python3-pecan-1.3.2-9.el8.noarch.rpm 823 kB/s | 283 kB 00:00 2023-11-16T23:47:50.047 INFO:teuthology.orchestra.run.smithi173.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 219 kB/s | 24 kB 00:00 2023-11-16T23:47:50.071 INFO:teuthology.orchestra.run.smithi173.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 249 kB/s | 29 kB 00:00 2023-11-16T23:47:50.097 INFO:teuthology.orchestra.run.smithi173.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2023-11-16T23:47:50.156 INFO:teuthology.orchestra.run.smithi173.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.3 MB/s | 255 kB 00:00 2023-11-16T23:47:50.181 INFO:teuthology.orchestra.run.smithi173.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-11-16T23:47:50.206 INFO:teuthology.orchestra.run.smithi173.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 563 kB/s | 61 kB 00:00 2023-11-16T23:47:50.256 INFO:teuthology.orchestra.run.smithi173.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 851 kB/s | 85 kB 00:00 2023-11-16T23:47:50.261 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-11-16T23:47:50.277 INFO:teuthology.orchestra.run.smithi023.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3906.g1c956 4.1 MB/s | 1.7 MB 00:00 2023-11-16T23:47:50.282 INFO:teuthology.orchestra.run.smithi173.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T23:47:50.307 INFO:teuthology.orchestra.run.smithi173.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 182 kB/s | 23 kB 00:00 2023-11-16T23:47:50.381 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-11-16T23:47:50.381 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-11-16T23:47:50.382 INFO:teuthology.orchestra.run.smithi173.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 466 kB/s | 47 kB 00:00 2023-11-16T23:47:50.386 INFO:teuthology.orchestra.run.smithi023.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3906.g1c 2.2 MB/s | 240 kB 00:00 2023-11-16T23:47:50.408 INFO:teuthology.orchestra.run.smithi173.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.9 MB/s | 200 kB 00:00 2023-11-16T23:47:50.445 INFO:teuthology.orchestra.run.smithi023.stdout:(32/131): ceph-mgr-rook-18.0.0-3906.g1c956514.e 817 kB/s | 47 kB 00:00 2023-11-16T23:47:50.466 INFO:teuthology.orchestra.run.smithi173.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 510 kB/s | 43 kB 00:00 2023-11-16T23:47:50.491 INFO:teuthology.orchestra.run.smithi173.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-11-16T23:47:50.495 INFO:teuthology.orchestra.run.smithi023.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3906.g1 290 kB/s | 15 kB 00:00 2023-11-16T23:47:50.517 INFO:teuthology.orchestra.run.smithi173.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 955 kB/s | 48 kB 00:00 2023-11-16T23:47:50.542 INFO:teuthology.orchestra.run.smithi173.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-11-16T23:47:50.568 INFO:teuthology.orchestra.run.smithi173.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 850 kB/s | 43 kB 00:00 2023-11-16T23:47:50.593 INFO:teuthology.orchestra.run.smithi173.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-11-16T23:47:50.604 INFO:teuthology.orchestra.run.smithi023.stdout:(34/131): ceph-volume-18.0.0-3906.g1c956514.el8 2.4 MB/s | 265 kB 00:00 2023-11-16T23:47:50.618 INFO:teuthology.orchestra.run.smithi173.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-11-16T23:47:50.660 INFO:teuthology.orchestra.run.smithi173.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-11-16T23:47:50.679 INFO:teuthology.orchestra.run.smithi023.stdout:(35/131): cephadm-18.0.0-3906.g1c956514.el8.noa 1.1 MB/s | 83 kB 00:00 2023-11-16T23:47:50.685 INFO:teuthology.orchestra.run.smithi173.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 280 kB/s | 19 kB 00:00 2023-11-16T23:47:50.711 INFO:teuthology.orchestra.run.smithi173.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-11-16T23:47:50.744 INFO:teuthology.orchestra.run.smithi173.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 9.3 MB/s | 559 kB 00:00 2023-11-16T23:47:50.769 INFO:teuthology.orchestra.run.smithi173.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 305 kB/s | 18 kB 00:00 2023-11-16T23:47:50.812 INFO:teuthology.orchestra.run.smithi173.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2023-11-16T23:47:50.813 INFO:teuthology.orchestra.run.smithi023.stdout:(36/131): ceph-mgr-diskprediction-local-18.0.0- 8.2 MB/s | 7.4 MB 00:00 2023-11-16T23:47:50.837 INFO:teuthology.orchestra.run.smithi173.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-11-16T23:47:50.838 INFO:teuthology.orchestra.run.smithi023.stdout:(37/131): cephfs-top-18.0.0-3906.g1c956514.el8. 113 kB/s | 18 kB 00:00 2023-11-16T23:47:50.896 INFO:teuthology.orchestra.run.smithi173.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-11-16T23:47:50.946 INFO:teuthology.orchestra.run.smithi173.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 17 MB/s | 2.3 MB 00:00 2023-11-16T23:47:50.971 INFO:teuthology.orchestra.run.smithi173.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-11-16T23:47:50.988 INFO:teuthology.orchestra.run.smithi173.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-11-16T23:47:51.014 INFO:teuthology.orchestra.run.smithi173.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-11-16T23:47:51.033 INFO:teuthology.orchestra.run.smithi173.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2023-11-16T23:47:51.060 INFO:teuthology.orchestra.run.smithi173.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-11-16T23:47:51.210 INFO:teuthology.orchestra.run.smithi023.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 221 kB/s | 59 kB 00:00 2023-11-16T23:47:51.210 INFO:teuthology.orchestra.run.smithi173.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 13 MB/s | 644 kB 00:00 2023-11-16T23:47:51.211 INFO:teuthology.orchestra.run.smithi173.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.9 MB/s | 171 kB 00:00 2023-11-16T23:47:51.211 INFO:teuthology.orchestra.run.smithi173.stdout:(96/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-11-16T23:47:51.211 INFO:teuthology.orchestra.run.smithi173.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-11-16T23:47:51.211 INFO:teuthology.orchestra.run.smithi173.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-11-16T23:47:51.215 INFO:teuthology.orchestra.run.smithi023.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2023-11-16T23:47:51.248 INFO:teuthology.orchestra.run.smithi023.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 705 kB/s | 306 kB 00:00 2023-11-16T23:47:51.249 INFO:teuthology.orchestra.run.smithi173.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-11-16T23:47:51.314 INFO:teuthology.orchestra.run.smithi023.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 656 kB/s | 44 kB 00:00 2023-11-16T23:47:51.325 INFO:teuthology.orchestra.run.smithi173.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-11-16T23:47:51.375 INFO:teuthology.orchestra.run.smithi173.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 778 kB/s | 39 kB 00:00 2023-11-16T23:47:51.425 INFO:teuthology.orchestra.run.smithi173.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-11-16T23:47:51.468 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-11-16T23:47:51.472 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-11-16T23:47:51.475 INFO:teuthology.orchestra.run.smithi173.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 11 MB/s | 538 kB 00:00 2023-11-16T23:47:51.521 INFO:teuthology.orchestra.run.smithi173.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-11-16T23:47:51.660 INFO:teuthology.orchestra.run.smithi173.stdout:(105/131): python3-scipy-1.0.0-21.module+el8.5. 25 MB/s | 14 MB 00:00 2023-11-16T23:47:51.721 INFO:teuthology.orchestra.run.smithi173.stdout:(106/131): thrift-0.13.0-2.el8.x86_64.rpm 1.1 MB/s | 1.7 MB 00:01 2023-11-16T23:47:51.752 INFO:teuthology.orchestra.run.smithi173.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 802 kB/s | 187 kB 00:00 2023-11-16T23:47:51.773 INFO:teuthology.orchestra.run.smithi023.stdout:(42/131): ceph-test-18.0.0-3906.g1c956514.el8.x 7.5 MB/s | 41 MB 00:05 2023-11-16T23:47:51.793 INFO:teuthology.orchestra.run.smithi173.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-11-16T23:47:51.798 INFO:teuthology.orchestra.run.smithi023.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 383 kB/s | 185 kB 00:00 2023-11-16T23:47:51.819 INFO:teuthology.orchestra.run.smithi173.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 434 kB/s | 27 kB 00:00 2023-11-16T23:47:51.832 INFO:teuthology.orchestra.run.smithi023.stdout:(44/131): python3-asyncssh-2.7.0-2.el8.noarch.r 770 kB/s | 476 kB 00:00 2023-11-16T23:47:51.856 INFO:teuthology.orchestra.run.smithi173.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2023-11-16T23:47:51.877 INFO:teuthology.orchestra.run.smithi023.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 216 kB/s | 16 kB 00:00 2023-11-16T23:47:51.878 INFO:teuthology.orchestra.run.smithi173.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 753 kB/s | 44 kB 00:00 2023-11-16T23:47:51.928 INFO:teuthology.orchestra.run.smithi173.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-11-16T23:47:51.970 INFO:teuthology.orchestra.run.smithi173.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-11-16T23:47:52.000 INFO:teuthology.orchestra.run.smithi023.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2023-11-16T23:47:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-11-16T23:47:52.041 INFO:teuthology.orchestra.run.smithi023.stdout:(47/131): python3-cachetools-3.1.1-4.el8.noarch 123 kB/s | 33 kB 00:00 2023-11-16T23:47:52.062 INFO:teuthology.orchestra.run.smithi173.stdout:(115/131): python3-numpy-1.14.3-10.el8.x86_64.r 18 MB/s | 3.7 MB 00:00 2023-11-16T23:47:52.088 INFO:teuthology.orchestra.run.smithi173.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.6 MB/s | 126 kB 00:00 2023-11-16T23:47:52.108 INFO:teuthology.orchestra.run.smithi023.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-11-16T23:47:52.113 INFO:teuthology.orchestra.run.smithi173.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-11-16T23:47:52.134 INFO:teuthology.orchestra.run.smithi023.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 608 kB/s | 81 kB 00:00 2023-11-16T23:47:52.175 INFO:teuthology.orchestra.run.smithi023.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2023-11-16T23:47:52.209 INFO:teuthology.orchestra.run.smithi023.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.1 MB/s | 384 kB 00:00 2023-11-16T23:47:52.243 INFO:teuthology.orchestra.run.smithi023.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 728 kB/s | 49 kB 00:00 2023-11-16T23:47:52.268 INFO:teuthology.orchestra.run.smithi023.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-11-16T23:47:52.293 INFO:teuthology.orchestra.run.smithi023.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-11-16T23:47:52.330 INFO:teuthology.orchestra.run.smithi173.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 428 kB/s | 103 kB 00:00 2023-11-16T23:47:52.335 INFO:teuthology.orchestra.run.smithi023.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 375 kB/s | 16 kB 00:00 2023-11-16T23:47:52.411 INFO:teuthology.orchestra.run.smithi023.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 450 kB/s | 34 kB 00:00 2023-11-16T23:47:52.414 INFO:teuthology.orchestra.run.smithi173.stdout:(119/131): python3-babel-2.5.1-7.el8.noarch.rpm 6.9 MB/s | 4.8 MB 00:00 2023-11-16T23:47:52.436 INFO:teuthology.orchestra.run.smithi023.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-16T23:47:52.447 INFO:teuthology.orchestra.run.smithi173.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 2.4 MB/s | 709 kB 00:00 2023-11-16T23:47:52.481 INFO:teuthology.orchestra.run.smithi173.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2023-11-16T23:47:52.486 INFO:teuthology.orchestra.run.smithi023.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-11-16T23:47:52.511 INFO:teuthology.orchestra.run.smithi023.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-11-16T23:47:52.523 INFO:teuthology.orchestra.run.smithi173.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-11-16T23:47:52.537 INFO:teuthology.orchestra.run.smithi023.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2023-11-16T23:47:52.562 INFO:teuthology.orchestra.run.smithi023.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2023-11-16T23:47:52.565 INFO:teuthology.orchestra.run.smithi173.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-11-16T23:47:52.587 INFO:teuthology.orchestra.run.smithi023.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2023-11-16T23:47:52.621 INFO:teuthology.orchestra.run.smithi023.stdout:(63/131): python3-kubernetes-11.0.0-6.el8.noarc 2.4 MB/s | 1.2 MB 00:00 2023-11-16T23:47:52.638 INFO:teuthology.orchestra.run.smithi023.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2023-11-16T23:47:52.674 INFO:teuthology.orchestra.run.smithi173.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 14 MB/s | 4.9 MB 00:00 2023-11-16T23:47:52.680 INFO:teuthology.orchestra.run.smithi023.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-11-16T23:47:52.705 INFO:teuthology.orchestra.run.smithi023.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2023-11-16T23:47:52.739 INFO:teuthology.orchestra.run.smithi023.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.1 MB/s | 251 kB 00:00 2023-11-16T23:47:52.740 INFO:teuthology.orchestra.run.smithi173.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-11-16T23:47:52.764 INFO:teuthology.orchestra.run.smithi023.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-11-16T23:47:52.781 INFO:teuthology.orchestra.run.smithi023.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2023-11-16T23:47:52.790 INFO:teuthology.orchestra.run.smithi173.stdout:(126/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-11-16T23:47:52.806 INFO:teuthology.orchestra.run.smithi023.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-11-16T23:47:52.857 INFO:teuthology.orchestra.run.smithi023.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 622 kB/s | 47 kB 00:00 2023-11-16T23:47:52.857 INFO:teuthology.orchestra.run.smithi173.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-11-16T23:47:52.882 INFO:teuthology.orchestra.run.smithi023.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.9 MB/s | 200 kB 00:00 2023-11-16T23:47:52.907 INFO:teuthology.orchestra.run.smithi023.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 854 kB/s | 43 kB 00:00 2023-11-16T23:47:52.932 INFO:teuthology.orchestra.run.smithi023.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-11-16T23:47:52.941 INFO:teuthology.orchestra.run.smithi173.stdout:(128/131): postgresql-server-10.21-2.module+el8 13 MB/s | 5.1 MB 00:00 2023-11-16T23:47:53.000 INFO:teuthology.orchestra.run.smithi173.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-11-16T23:47:53.028 INFO:teuthology.orchestra.run.smithi023.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 398 kB/s | 48 kB 00:00 2023-11-16T23:47:53.058 INFO:teuthology.orchestra.run.smithi023.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 695 kB/s | 87 kB 00:00 2023-11-16T23:47:53.084 INFO:teuthology.orchestra.run.smithi023.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 780 kB/s | 43 kB 00:00 2023-11-16T23:47:53.109 INFO:teuthology.orchestra.run.smithi023.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-11-16T23:47:53.143 INFO:teuthology.orchestra.run.smithi023.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2023-11-16T23:47:53.176 INFO:teuthology.orchestra.run.smithi023.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-11-16T23:47:53.202 INFO:teuthology.orchestra.run.smithi023.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 319 kB/s | 19 kB 00:00 2023-11-16T23:47:53.227 INFO:teuthology.orchestra.run.smithi023.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-11-16T23:47:53.269 INFO:teuthology.orchestra.run.smithi023.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 8.2 MB/s | 559 kB 00:00 2023-11-16T23:47:53.294 INFO:teuthology.orchestra.run.smithi023.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 268 kB/s | 18 kB 00:00 2023-11-16T23:47:53.328 INFO:teuthology.orchestra.run.smithi023.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-11-16T23:47:53.361 INFO:teuthology.orchestra.run.smithi023.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-11-16T23:47:53.392 INFO:teuthology.orchestra.run.smithi173.stdout:(130/131): librados2-18.0.0-3906.g1c956514.el8. 6.1 MB/s | 3.7 MB 00:00 2023-11-16T23:47:53.403 INFO:teuthology.orchestra.run.smithi023.stdout:(87/131): thrift-0.13.0-2.el8.x86_64.rpm 2.6 MB/s | 1.7 MB 00:00 2023-11-16T23:47:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.1 MB/s | 72 kB 00:00 2023-11-16T23:47:53.482 INFO:teuthology.orchestra.run.smithi023.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-11-16T23:47:53.483 INFO:teuthology.orchestra.run.smithi173.stdout:(131/131): librbd1-18.0.0-3906.g1c956514.el8.x8 5.7 MB/s | 3.6 MB 00:00 2023-11-16T23:47:53.489 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:47:53.489 INFO:teuthology.orchestra.run.smithi173.stdout:Total 13 MB/s | 197 MB 00:15 2023-11-16T23:47:53.504 INFO:teuthology.orchestra.run.smithi023.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 825 kB/s | 83 kB 00:00 2023-11-16T23:47:53.529 INFO:teuthology.orchestra.run.smithi023.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-11-16T23:47:53.555 INFO:teuthology.orchestra.run.smithi023.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-11-16T23:47:53.580 INFO:teuthology.orchestra.run.smithi023.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 772 kB/s | 39 kB 00:00 2023-11-16T23:47:53.622 INFO:teuthology.orchestra.run.smithi023.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2023-11-16T23:47:53.673 INFO:teuthology.orchestra.run.smithi023.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.8 MB/s | 171 kB 00:00 2023-11-16T23:47:53.706 INFO:teuthology.orchestra.run.smithi023.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.6 MB/s | 138 kB 00:00 2023-11-16T23:47:53.748 INFO:teuthology.orchestra.run.smithi023.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 5.5 MB/s | 2.3 MB 00:00 2023-11-16T23:47:53.782 INFO:teuthology.orchestra.run.smithi023.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-11-16T23:47:53.807 INFO:teuthology.orchestra.run.smithi023.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 924 kB/s | 54 kB 00:00 2023-11-16T23:47:53.840 INFO:teuthology.orchestra.run.smithi023.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.4 MB/s | 259 kB 00:00 2023-11-16T23:47:53.874 INFO:teuthology.orchestra.run.smithi023.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-11-16T23:47:53.899 INFO:teuthology.orchestra.run.smithi023.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 663 kB/s | 39 kB 00:00 2023-11-16T23:47:53.941 INFO:teuthology.orchestra.run.smithi023.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-11-16T23:47:53.992 INFO:teuthology.orchestra.run.smithi023.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 5.7 MB/s | 538 kB 00:00 2023-11-16T23:47:54.017 INFO:teuthology.orchestra.run.smithi023.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-11-16T23:47:54.051 INFO:teuthology.orchestra.run.smithi023.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2023-11-16T23:47:54.092 INFO:teuthology.orchestra.run.smithi023.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-11-16T23:47:54.126 INFO:teuthology.orchestra.run.smithi023.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 797 kB/s | 27 kB 00:00 2023-11-16T23:47:54.216 INFO:teuthology.orchestra.run.smithi023.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2023-11-16T23:47:54.221 INFO:teuthology.orchestra.run.smithi023.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-11-16T23:47:54.332 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-11-16T23:47:54.360 INFO:teuthology.orchestra.run.smithi023.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-11-16T23:47:54.411 INFO:teuthology.orchestra.run.smithi023.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-11-16T23:47:54.453 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-11-16T23:47:54.453 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-11-16T23:47:54.494 INFO:teuthology.orchestra.run.smithi023.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2023-11-16T23:47:54.519 INFO:teuthology.orchestra.run.smithi023.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.0 MB/s | 225 kB 00:00 2023-11-16T23:47:54.545 INFO:teuthology.orchestra.run.smithi023.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2023-11-16T23:47:54.570 INFO:teuthology.orchestra.run.smithi023.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-11-16T23:47:54.604 INFO:teuthology.orchestra.run.smithi023.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 340 kB/s | 20 kB 00:00 2023-11-16T23:47:54.629 INFO:teuthology.orchestra.run.smithi023.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.7 MB/s | 103 kB 00:00 2023-11-16T23:47:54.671 INFO:teuthology.orchestra.run.smithi023.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-11-16T23:47:54.721 INFO:teuthology.orchestra.run.smithi023.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-11-16T23:47:54.905 INFO:teuthology.orchestra.run.smithi023.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 11 MB/s | 14 MB 00:01 2023-11-16T23:47:54.964 INFO:teuthology.orchestra.run.smithi023.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-11-16T23:47:55.014 INFO:teuthology.orchestra.run.smithi023.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-11-16T23:47:55.081 INFO:teuthology.orchestra.run.smithi023.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 11 MB/s | 4.9 MB 00:00 2023-11-16T23:47:55.148 INFO:teuthology.orchestra.run.smithi023.stdout:(125/131): postgresql-contrib-10.21-2.module+el 12 MB/s | 808 kB 00:00 2023-11-16T23:47:55.215 INFO:teuthology.orchestra.run.smithi023.stdout:(126/131): openblas-0.3.15-3.el8.x86_64.rpm 9.6 MB/s | 4.7 MB 00:00 2023-11-16T23:47:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-11-16T23:47:55.374 INFO:teuthology.orchestra.run.smithi023.stdout:(128/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-11-16T23:47:55.441 INFO:teuthology.orchestra.run.smithi023.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-11-16T23:47:55.537 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-11-16T23:48:02.514 INFO:teuthology.orchestra.run.smithi023.stdout:(130/131): librados2-18.0.0-3906.g1c956514.el8. 7.0 MB/s | 3.7 MB 00:00 2023-11-16T23:48:02.514 INFO:teuthology.orchestra.run.smithi023.stdout:(131/131): librbd1-18.0.0-3906.g1c956514.el8.x8 4.0 MB/s | 3.6 MB 00:00 2023-11-16T23:48:02.514 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:48:02.514 INFO:teuthology.orchestra.run.smithi023.stdout:Total 12 MB/s | 197 MB 00:16 2023-11-16T23:48:02.515 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-11-16T23:48:02.550 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-11-16T23:48:02.550 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-11-16T23:48:02.550 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-11-16T23:48:02.551 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-11-16T23:48:02.551 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T23:48:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T23:48:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T23:48:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 5/134 2023-11-16T23:48:02.556 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-11-16T23:48:02.556 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T23:48:02.556 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T23:48:02.557 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T23:48:02.557 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T23:48:02.557 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T23:48:02.586 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T23:48:02.586 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T23:48:02.587 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:02.587 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:02.587 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:02.587 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:02.587 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 9/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T23:48:02.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T23:48:02.589 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T23:48:02.590 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T23:48:02.590 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T23:48:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 5/134 2023-11-16T23:48:02.720 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T23:48:02.860 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:03.268 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:03.306 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T23:48:03.443 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:03.520 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T23:48:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:48:03.924 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-16T23:48:04.194 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T23:48:04.233 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T23:48:04.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T23:48:04.484 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T23:48:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-11-16T23:48:04.730 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T23:48:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T23:48:04.835 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T23:48:04.923 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T23:48:05.055 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T23:48:05.120 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T23:48:05.150 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T23:48:05.251 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:05.319 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T23:48:05.344 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:05.350 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T23:48:05.487 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-16T23:48:05.493 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:05.604 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T23:48:05.620 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:05.694 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T23:48:05.809 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 26/134 2023-11-16T23:48:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-16T23:48:05.948 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 27/134 2023-11-16T23:48:06.049 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-16T23:48:06.177 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-16T23:48:06.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-16T23:48:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T23:48:06.369 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 5/134 2023-11-16T23:48:06.504 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T23:48:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T23:48:06.841 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 5/134 2023-11-16T23:48:06.912 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 28/134 2023-11-16T23:48:07.027 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:07.179 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:48:07.234 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T23:48:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T23:48:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:07.488 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:48:07.509 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T23:48:07.513 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T23:48:07.666 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:48:07.789 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T23:48:07.880 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-16T23:48:08.003 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T23:48:08.109 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T23:48:08.150 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T23:48:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T23:48:08.713 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T23:48:09.041 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T23:48:09.143 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T23:48:09.274 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-16T23:48:09.401 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T23:48:09.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-16T23:48:09.560 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:09.687 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 14/134 2023-11-16T23:48:09.726 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T23:48:09.769 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T23:48:09.843 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:09.890 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 26/134 2023-11-16T23:48:10.024 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 27/134 2023-11-16T23:48:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 15/134 2023-11-16T23:48:10.078 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T23:48:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-16T23:48:10.279 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 28/134 2023-11-16T23:48:10.305 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T23:48:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-16T23:48:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T23:48:10.529 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T23:48:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-16T23:48:10.766 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 36/134 2023-11-16T23:48:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T23:48:10.956 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 37/134 2023-11-16T23:48:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-16T23:48:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T23:48:11.151 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T23:48:11.328 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T23:48:11.563 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T23:48:11.714 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T23:48:11.900 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-16T23:48:12.069 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T23:48:12.084 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T23:48:12.261 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T23:48:12.458 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T23:48:12.780 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-16T23:48:12.861 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T23:48:12.960 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T23:48:13.188 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T23:48:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-16T23:48:13.388 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T23:48:13.635 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T23:48:13.672 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-16T23:48:13.833 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-16T23:48:13.898 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T23:48:13.960 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-16T23:48:14.076 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-16T23:48:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-16T23:48:14.203 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T23:48:14.289 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-16T23:48:14.434 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T23:48:14.444 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T23:48:14.565 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T23:48:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T23:48:14.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T23:48:14.742 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 26/134 2023-11-16T23:48:14.855 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 36/134 2023-11-16T23:48:14.856 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T23:48:14.890 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 27/134 2023-11-16T23:48:15.014 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 37/134 2023-11-16T23:48:15.081 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T23:48:15.151 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 28/134 2023-11-16T23:48:15.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-16T23:48:15.246 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T23:48:15.342 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-16T23:48:15.351 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T23:48:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T23:48:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T23:48:15.601 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T23:48:15.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-16T23:48:15.883 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T23:48:16.053 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-16T23:48:16.172 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T23:48:16.208 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T23:48:16.288 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T23:48:16.392 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T23:48:16.528 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-16T23:48:16.600 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T23:48:16.695 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-16T23:48:16.756 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T23:48:16.898 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-16T23:48:16.976 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T23:48:17.048 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T23:48:17.075 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-16T23:48:17.243 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-16T23:48:17.296 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T23:48:17.436 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-16T23:48:17.505 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T23:48:17.515 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-16T23:48:17.639 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-16T23:48:17.676 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T23:48:17.759 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T23:48:17.863 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-16T23:48:17.974 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T23:48:18.170 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T23:48:18.220 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-16T23:48:18.327 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T23:48:18.517 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-16T23:48:18.540 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T23:48:18.693 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-16T23:48:18.796 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T23:48:18.959 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-16T23:48:19.033 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T23:48:19.181 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T23:48:19.233 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-16T23:48:19.362 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T23:48:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-16T23:48:19.503 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-16T23:48:19.553 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T23:48:19.563 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-16T23:48:19.776 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-16T23:48:19.798 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T23:48:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-16T23:48:20.007 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T23:48:20.012 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T23:48:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-16T23:48:20.162 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T23:48:20.233 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T23:48:20.290 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T23:48:20.308 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-16T23:48:20.433 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 36/134 2023-11-16T23:48:20.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-16T23:48:20.547 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T23:48:20.617 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 37/134 2023-11-16T23:48:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-16T23:48:20.768 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-16T23:48:20.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T23:48:20.901 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-16T23:48:20.940 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-16T23:48:20.998 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T23:48:21.141 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T23:48:21.196 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-16T23:48:21.235 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-16T23:48:21.265 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T23:48:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-16T23:48:21.483 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T23:48:21.498 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-16T23:48:21.676 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T23:48:21.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-16T23:48:21.767 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-16T23:48:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T23:48:21.916 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T23:48:21.941 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-16T23:48:22.022 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-16T23:48:22.081 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T23:48:22.276 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T23:48:22.280 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-16T23:48:22.455 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-16T23:48:22.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T23:48:22.618 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-16T23:48:22.763 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T23:48:22.805 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-16T23:48:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T23:48:23.003 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-16T23:48:23.024 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T23:48:23.083 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T23:48:23.182 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-16T23:48:23.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T23:48:23.312 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T23:48:23.375 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-16T23:48:23.526 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T23:48:23.566 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T23:48:23.626 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-16T23:48:23.716 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T23:48:23.876 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T23:48:23.926 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T23:48:24.060 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-16T23:48:24.087 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T23:48:24.169 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T23:48:24.353 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T23:48:24.409 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T23:48:24.424 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-16T23:48:24.627 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T23:48:24.632 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-16T23:48:24.821 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T23:48:24.916 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-16T23:48:25.054 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T23:48:25.169 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-16T23:48:25.213 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T23:48:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T23:48:25.408 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:25.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T23:48:25.491 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:25.499 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-16T23:48:25.667 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 93/134 2023-11-16T23:48:25.708 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T23:48:25.709 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-16T23:48:25.903 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-16T23:48:26.171 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-16T23:48:26.375 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-16T23:48:26.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-16T23:48:26.833 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T23:48:26.923 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-16T23:48:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T23:48:27.217 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-16T23:48:27.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T23:48:27.492 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-16T23:48:27.690 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T23:48:27.703 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-16T23:48:27.876 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T23:48:27.976 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T23:48:28.083 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T23:48:28.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T23:48:28.354 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T23:48:28.549 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T23:48:28.729 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T23:48:28.937 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T23:48:28.972 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T23:48:29.131 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T23:48:29.147 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:29.218 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T23:48:29.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 93/134 2023-11-16T23:48:29.461 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:29.556 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T23:48:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T23:48:29.890 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T23:48:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T23:48:30.295 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T23:48:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T23:48:30.606 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:30.690 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T23:48:30.874 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:48:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T23:48:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T23:48:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T23:48:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T23:48:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T23:48:33.174 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:33.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T23:48:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T23:48:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:34.113 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T23:48:34.237 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:34.321 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:34.332 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T23:48:34.480 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:48:35.210 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T23:48:35.389 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:35.485 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 92/134 2023-11-16T23:48:35.673 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 93/134 2023-11-16T23:48:40.767 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:40.990 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 94/134 2023-11-16T23:48:41.988 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:42.137 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 95/134 2023-11-16T23:48:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:48:49.648 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:48:49.648 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2023-11-16T23:48:49.648 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /run 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-11-16T23:48:49.649 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:48:50.030 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:48:51.598 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:48:51.811 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:48:52.030 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:48:52.254 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:48:52.705 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:48:52.913 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:48:52.930 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:48:53.556 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:48:53.578 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2023-11-16T23:48:53.578 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2023-11-16T23:48:53.578 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /run 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2023-11-16T23:48:53.579 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:48:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:48:54.613 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:48:54.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:48:54.959 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 102/134 2023-11-16T23:48:55.210 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 103/134 2023-11-16T23:48:55.395 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:48:55.479 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:48:55.608 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:48:55.650 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:48:55.868 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:48:55.868 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:48:56.027 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:48:56.477 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:48:56.685 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:48:56.702 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:48:56.831 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:48:56.995 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:48:57.087 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:48:57.462 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 107/134 2023-11-16T23:48:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:48:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:48:57.928 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:48:57.940 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:48:58.370 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:48:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:48:58.758 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 102/134 2023-11-16T23:48:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 103/134 2023-11-16T23:48:59.184 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:48:59.203 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 110/134 2023-11-16T23:48:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:48:59.416 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:48:59.452 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:48:59.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:48:59.686 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:48:59.725 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:48:59.903 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:00.248 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:00.419 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:00.682 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:49:00.876 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:49:00.967 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:49:01.311 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 107/134 2023-11-16T23:49:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:49:01.574 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:49:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:49:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:49:03.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 110/134 2023-11-16T23:49:03.250 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:49:03.284 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:49:03.476 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:49:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 96/134 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2023-11-16T23:49:03.536 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2023-11-16T23:49:03.537 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2023-11-16T23:49:03.537 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2023-11-16T23:49:03.537 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-11-16T23:49:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:49:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:49:04.012 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:04.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:04.360 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:04.667 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 115/134 2023-11-16T23:49:04.852 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T23:49:05.054 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T23:49:05.733 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 97/134 2023-11-16T23:49:05.902 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:49:05.967 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T23:49:06.182 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 98/134 2023-11-16T23:49:06.309 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T23:49:06.344 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:49:06.504 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 119/134 2023-11-16T23:49:06.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T23:49:06.891 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 99/134 2023-11-16T23:49:06.897 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 121/134 2023-11-16T23:49:07.069 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T23:49:07.085 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:49:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 100/134 2023-11-16T23:49:08.081 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 115/134 2023-11-16T23:49:09.272 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-16T23:49:09.393 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:09.461 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 117/134 2023-11-16T23:49:09.659 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:09.678 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:09.749 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-16T23:49:09.877 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 124/134 2023-11-16T23:49:09.933 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T23:49:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:49:10.123 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T23:49:10.196 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 125/134 2023-11-16T23:49:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 101/134 2023-11-16T23:49:10.354 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-16T23:49:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 102/134 2023-11-16T23:49:10.492 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 126/134 2023-11-16T23:49:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 121/134 2023-11-16T23:49:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 103/134 2023-11-16T23:49:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T23:49:10.794 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T23:49:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:49:10.976 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T23:49:10.985 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T23:49:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 104/134 2023-11-16T23:49:11.164 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T23:49:11.375 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:49:11.392 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T23:49:11.474 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T23:49:11.717 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T23:49:11.730 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T23:49:11.730 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:11.929 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:11.929 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:12.193 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:12.441 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:12.531 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 105/134 2023-11-16T23:49:12.751 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:49:12.885 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 106/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 1/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 3/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 4/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 5/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:49:12.957 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 9/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 10/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 11/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 12/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 13/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 14/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 15/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 16/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 17/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 18/134 2023-11-16T23:49:12.958 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 19/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 20/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 21/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 22/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 23/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 24/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 25/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 26/134 2023-11-16T23:49:12.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 27/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 28/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 29/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 30/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 31/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 32/134 2023-11-16T23:49:12.968 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 33/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 34/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 35/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 36/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 37/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 38/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T23:49:12.969 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T23:49:12.970 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T23:49:12.971 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T23:49:12.972 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T23:49:12.973 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T23:49:12.974 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T23:49:12.975 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 129/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 131/134 2023-11-16T23:49:12.976 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:12.977 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T23:49:13.031 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:13.274 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 107/134 2023-11-16T23:49:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:49:13.524 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 124/134 2023-11-16T23:49:13.580 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 108/134 2023-11-16T23:49:13.776 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:49:13.782 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 125/134 2023-11-16T23:49:13.805 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 109/134 2023-11-16T23:49:14.118 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 126/134 2023-11-16T23:49:14.495 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T23:49:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T23:49:14.755 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T23:49:14.949 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T23:49:14.962 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T23:49:15.152 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T23:49:15.161 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T23:49:15.239 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 110/134 2023-11-16T23:49:15.292 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-16T23:49:15.292 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:15.483 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:49:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:15.509 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:52 ago on Thu 16 Nov 2023 11:47:23 PM UTC. 2023-11-16T23:49:15.548 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 111/134 2023-11-16T23:49:15.709 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:49:15.754 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:15.782 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3906.g1c956 112/134 2023-11-16T23:49:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:15.984 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:16.412 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 113/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 1/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 3/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 4/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 5/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:49:16.467 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 9/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 10/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 11/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 12/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 13/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 14/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 15/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 16/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 17/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 18/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 19/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 20/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 21/134 2023-11-16T23:49:16.468 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 22/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 23/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 24/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 25/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 26/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 27/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 28/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 29/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 30/134 2023-11-16T23:49:16.469 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 31/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 32/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 33/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 34/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 35/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 36/134 2023-11-16T23:49:16.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 37/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 38/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T23:49:16.479 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T23:49:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T23:49:16.481 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T23:49:16.481 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T23:49:16.481 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T23:49:16.481 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T23:49:16.481 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-16T23:49:16.482 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-16T23:49:16.483 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-16T23:49:16.484 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-16T23:49:16.485 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-16T23:49:16.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-16T23:49:16.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 129/134 2023-11-16T23:49:16.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T23:49:16.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 131/134 2023-11-16T23:49:16.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:16.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T23:49:16.609 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:18.191 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-11-16T23:49:18.410 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.411 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.412 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T23:49:18.413 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-3.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T23:49:18.414 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T23:49:18.415 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T23:49:18.416 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T23:49:18.417 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T23:49:18.418 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T23:49:18.419 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.420 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-16T23:49:18.421 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-11-16T23:49:18.646 DEBUG:teuthology.parallel:result is None 2023-11-16T23:49:19.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:19.030 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:53 ago on Thu 16 Nov 2023 11:47:26 PM UTC. 2023-11-16T23:49:21.128 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 114/134 2023-11-16T23:49:21.751 INFO:teuthology.orchestra.run.smithi173.stdout:Installed products updated. 2023-11-16T23:49:21.962 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:49:21.962 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2023-11-16T23:49:21.962 INFO:teuthology.orchestra.run.smithi173.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T23:49:21.962 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.963 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.964 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-3.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.965 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T23:49:21.966 INFO:teuthology.orchestra.run.smithi173.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T23:49:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T23:49:21.968 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T23:49:21.969 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T23:49:21.970 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T23:49:21.971 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T23:49:21.972 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T23:49:21.973 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T23:49:21.973 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:49:21.973 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-11-16T23:49:22.160 DEBUG:teuthology.parallel:result is None 2023-11-16T23:49:22.410 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 115/134 2023-11-16T23:49:22.749 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-16T23:49:22.973 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 117/134 2023-11-16T23:49:23.251 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-16T23:49:23.424 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T23:49:23.698 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-16T23:49:23.921 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-16T23:49:24.105 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 121/134 2023-11-16T23:49:24.284 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T23:49:27.865 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:28.149 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:28.191 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T23:49:28.434 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 124/134 2023-11-16T23:49:28.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 125/134 2023-11-16T23:49:28.993 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 126/134 2023-11-16T23:49:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T23:49:29.573 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T23:49:29.689 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-16T23:49:29.882 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T23:49:29.911 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-16T23:49:30.140 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T23:49:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-16T23:49:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-16T23:49:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T23:49:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:31.286 INFO:teuthology.orchestra.run.smithi023.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-16T23:49:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:49:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:18.0.0-3906.g1c956514.el8.x86_64 1/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 3/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 4/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3906.g1c956 5/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 6/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 7/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 8/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 9/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 10/134 2023-11-16T23:49:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 11/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 12/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 13/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_ 14/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 15/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 16/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:18.0.0-3906.g1c956514.el8.x86_6 17/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86 18/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 19/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3906.g1c956514.el 20/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:18.0.0-3906.g1c956514.el8. 21/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_6 22/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 23/134 2023-11-16T23:49:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 24/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 25/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 26/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 27/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 28/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3906.g1c956514. 29/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noa 30/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.n 31/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c9 32/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el 33/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 34/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.e 35/134 2023-11-16T23:49:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 36/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:18.0.0-3906.g1c956514.el8.noarch 37/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 38/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T23:49:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T23:49:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T23:49:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T23:49:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T23:49:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T23:49:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T23:49:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T23:49:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-16T23:49:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-16T23:49:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-16T23:49:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-16T23:49:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-16T23:49:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-16T23:49:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:18.0.0-3906.g1c956514.el8.x86_64 129/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 131/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T23:49:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T23:49:34.616 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T23:49:34.616 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:08 ago on Thu 16 Nov 2023 11:47:26 PM UTC. 2023-11-16T23:49:39.787 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2023-11-16T23:49:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:49:40.028 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2023-11-16T23:49:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T23:49:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:18.0.0-3906.g1c956514.el8.noarch 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-3.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T23:49:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T23:49:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T23:49:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T23:49:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T23:49:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:18.0.0-3906.g1c956514.el8.x86_64 2023-11-16T23:49:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:49:40.039 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-11-16T23:49:40.489 DEBUG:teuthology.parallel:result is None 2023-11-16T23:49:40.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:40.738 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T23:49:40.792 INFO:teuthology.orchestra.run.smithi023.stdout:18.0.0-3906.g1c956514.el8 2023-11-16T23:49:40.793 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3906.g1c956514.el8 2023-11-16T23:49:40.793 INFO:teuthology.task.install:The correct ceph version 18.0.0-3906.g1c956514 is installed. 2023-11-16T23:49:40.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:41.008 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T23:49:41.038 INFO:teuthology.orchestra.run.smithi156.stdout:18.0.0-3906.g1c956514.el8 2023-11-16T23:49:41.038 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3906.g1c956514.el8 2023-11-16T23:49:41.038 INFO:teuthology.task.install:The correct ceph version 18.0.0-3906.g1c956514 is installed. 2023-11-16T23:49:41.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:41.188 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T23:49:41.217 INFO:teuthology.orchestra.run.smithi173.stdout:18.0.0-3906.g1c956514.el8 2023-11-16T23:49:41.217 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3906.g1c956514.el8 2023-11-16T23:49:41.217 INFO:teuthology.task.install:The correct ceph version 18.0.0-3906.g1c956514 is installed. 2023-11-16T23:49:41.218 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T23:49:41.218 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:49:41.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T23:49:41.314 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:49:41.314 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T23:49:41.364 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:49:41.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T23:49:41.418 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T23:49:41.418 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:49:41.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T23:49:41.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T23:49:41.596 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:49:41.596 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T23:49:41.628 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T23:49:41.699 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:49:41.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T23:49:41.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T23:49:41.802 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T23:49:41.802 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:49:41.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T23:49:41.871 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T23:49:41.980 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:49:41.980 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T23:49:42.013 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T23:49:42.086 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:49:42.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T23:49:42.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T23:49:42.187 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T23:49:42.187 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:49:42.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T23:49:42.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T23:49:42.374 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:49:42.374 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T23:49:42.407 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T23:49:42.478 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:49:42.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T23:49:42.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T23:49:42.583 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T23:49:42.667 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-11-16T23:49:42.667 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:42.668 INFO:tasks.cephadm:Cluster fsid is d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:49:42.668 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T23:49:42.668 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.156', 'mon.c': '172.21.15.173'} 2023-11-16T23:49:42.668 INFO:tasks.cephadm:First mon is mon.a on smithi023 2023-11-16T23:49:42.668 INFO:tasks.cephadm:First mgr is x 2023-11-16T23:49:42.668 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T23:49:42.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2023-11-16T23:49:42.745 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2023-11-16T23:49:42.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2023-11-16T23:49:42.818 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-11-16T23:49:42.818 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:43.047 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1c9565146c8647d4e00eb6c7ae626dc192680e7b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/74215/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3906-g1c956514', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3906.g1c956514'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-11-16 20:52:31.578811', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-11-16T23:49:43.193 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-choffman-62793, sha1 1c9565146c8647d4e00eb6c7ae626dc192680e7b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-16T23:49:43.193 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/x86_64/flavors/default/cephadm 2023-11-16T23:49:43.193 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:43.739 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 23:49 /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:43.739 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.104 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 23:49 /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.104 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/1c9565146c8647d4e00eb6c7ae626dc192680e7b/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.612 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 23:49 /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.613 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.664 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.686 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T23:49:44.730 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b on all hosts... 2023-11-16T23:49:44.730 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b pull 2023-11-16T23:49:44.735 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b pull 2023-11-16T23:49:44.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b pull 2023-11-16T23:49:46.328 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b... 2023-11-16T23:49:46.384 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b... 2023-11-16T23:49:48.990 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b... 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev)", 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "ace6904b066bfa00ae45df809dc67d592e0953ee68f7eed3c36f7d4e23e52db2", 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cb1d293acfc56ca3305212d5789a9da86fb3fb7a4af319d5f427084581ab00ca" 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-16T23:50:11.319 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev)", 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "ace6904b066bfa00ae45df809dc67d592e0953ee68f7eed3c36f7d4e23e52db2", 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cb1d293acfc56ca3305212d5789a9da86fb3fb7a4af319d5f427084581ab00ca" 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-16T23:50:17.076 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev)", 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "ace6904b066bfa00ae45df809dc67d592e0953ee68f7eed3c36f7d4e23e52db2", 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cb1d293acfc56ca3305212d5789a9da86fb3fb7a4af319d5f427084581ab00ca" 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-11-16T23:50:22.022 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-16T23:50:22.041 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2023-11-16T23:50:22.124 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2023-11-16T23:50:22.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2023-11-16T23:50:22.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2023-11-16T23:50:22.267 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2023-11-16T23:50:22.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2023-11-16T23:50:22.330 INFO:tasks.cephadm:Writing seed config... 2023-11-16T23:50:22.331 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T23:50:22.331 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T23:50:22.331 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T23:50:22.331 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T23:50:22.332 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-11-16T23:50:22.333 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T23:50:22.334 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-16T23:50:22.335 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-11-16T23:50:22.335 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:50:22.335 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T23:50:22.381 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d04064ea-84da-11ee-8dbe-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T23:50:22.381 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service 2023-11-16T23:50:22.426 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x.service 2023-11-16T23:50:22.472 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T23:50:22.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b -v bootstrap --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T23:50:22.495 INFO:journalctl@ceph.mon.a.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:50:22.541 INFO:journalctl@ceph.mgr.x.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:50:22.822 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-11-16T23:50:22.822 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b', '-v', 'bootstrap', '--fsid', 'd04064ea-84da-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2023-11-16T23:50:22.999 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.1.1 2023-11-16T23:50:23.000 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-11-16T23:50:23.000 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2023-11-16T23:50:23.176 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.1.1 2023-11-16T23:50:23.176 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2023-11-16T23:50:23.176 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2023-11-16T23:50:23.191 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T23:50:23.191 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T23:50:23.204 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T23:50:23.205 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:50:23.218 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2023-11-16T23:50:23.231 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2023-11-16T23:50:23.231 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2023-11-16T23:50:23.232 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2023-11-16T23:50:23.404 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.1.1 2023-11-16T23:50:23.405 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.1.1 is present 2023-11-16T23:50:23.405 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2023-11-16T23:50:23.405 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2023-11-16T23:50:23.418 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T23:50:23.418 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T23:50:23.431 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T23:50:23.432 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:50:23.446 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 139898051131880 on /run/cephadm/d04064ea-84da-11ee-8dbe-212e2dc638e7.lock 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 139898051131880 acquired on /run/cephadm/d04064ea-84da-11ee-8dbe-212e2dc638e7.lock 2023-11-16T23:50:23.460 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2023-11-16T23:50:23.461 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2023-11-16T23:50:23.461 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2023-11-16T23:50:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-11-16T23:50:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2023-11-16T23:50:23.476 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T23:50:23.476 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-16T23:50:23.476 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-11-16T23:50:23.483 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T23:50:23.484 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2023-11-16T23:50:23.484 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2023-11-16T23:50:23.484 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-16T23:50:23.485 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T23:50:23.486 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b... 2023-11-16T23:50:23.961 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout ace6904b066bfa00ae45df809dc67d592e0953ee68f7eed3c36f7d4e23e52db2 2023-11-16T23:50:23.962 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b... 2023-11-16T23:50:23.962 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2023-11-16T23:50:23.962 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:eb1dc24ed1b729f25bfe21673f4e5cc46a126932dc459841b1c31fbebaca605e 2023-11-16T23:50:23.962 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:837e66e3ff89804ed1bb42786f80e6c23534ff65fe48526d5007449f4e1f79b0 2023-11-16T23:50:23.962 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:ace6904b066bfa00ae45df809dc67d592e0953ee68f7eed3c36f7d4e23e52db2 2023-11-16T23:50:23.990 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2023-11-16T23:50:23.990 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Storing signatures 2023-11-16T23:50:24.941 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-16T23:50:24.942 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-16T23:50:24.942 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2023-11-16T23:50:26.324 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2023-11-16T23:50:26.324 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2023-11-16T23:50:27.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDCqlZlLBVpLBAAEGptGTXI1diX3B/N6QOqDQ== 2023-11-16T23:50:27.979 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDDqlZlOCToJBAAMFkrD3smF6RfHn5fsXfAew== 2023-11-16T23:50:28.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDEqlZl+O/4FhAAPrjUbuzPCtngnZeI9vTn6Q== 2023-11-16T23:50:28.757 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = pacific 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:50:29.636 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2023-11-16T23:50:30.520 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2023-11-16T23:50:30.750 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T23:50:30.963 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T23:50:31.189 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target. 2023-11-16T23:50:31.190 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target. 2023-11-16T23:50:31.489 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T23:50:31.490 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service: Unit ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service not loaded. 2023-11-16T23:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target.wants/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T23:50:31.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:31 smithi023 systemd[1]: Starting Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:50:32.501 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:32 smithi023 podman[97280]: 2023-11-16T23:50:32.502 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:32 smithi023 bash[97280]: 0a74aaea994ddf442ec8a1c59e3a9ac91e371df0e9f17178d55c4b638e3a099c 2023-11-16T23:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T23:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2023-11-16T23:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T23:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2023-11-16T23:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T23:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2023-11-16T23:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2023-11-16T23:50:32.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:32 smithi023 systemd[1]: Started Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:50:32.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:32 smithi023 ceph-mon[97320]: mkfs d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:32.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:32 smithi023 ceph-mon[97320]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T23:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.72157s) 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-11-16T23:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.145+0000 7fed4fd50700 1 Processor -- start 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.145+0000 7fed4fd50700 1 -- start start 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.146+0000 7fed4fd50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.146+0000 7fed4fd50700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48106260 con 0x7fed48105910 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.147+0000 7fed4daec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.147+0000 7fed4daec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58786/0 (socket says 172.21.15.23:58786) 2023-11-16T23:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.147+0000 7fed4daec700 1 -- 172.21.15.23:0/631456305 learned_addr learned my addr 172.21.15.23:0/631456305 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.147+0000 7fed4daec700 1 -- 172.21.15.23:0/631456305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed481063a0 con 0x7fed48105910 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.147+0000 7fed4daec700 1 --2- 172.21.15.23:0/631456305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed440098a0 tx=0x7fed44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6611a52c88d91cb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4caea700 1 -- 172.21.15.23:0/631456305 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed44004160 con 0x7fed48105910 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4caea700 1 -- 172.21.15.23:0/631456305 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed440042c0 con 0x7fed48105910 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4caea700 1 -- 172.21.15.23:0/631456305 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed440045a0 con 0x7fed48105910 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 -- 172.21.15.23:0/631456305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 msgr2=0x7fed48105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 --2- 172.21.15.23:0/631456305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed440098a0 tx=0x7fed44009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 -- 172.21.15.23:0/631456305 shutdown_connections 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 --2- 172.21.15.23:0/631456305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed48105d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 -- 172.21.15.23:0/631456305 >> 172.21.15.23:0/631456305 conn(0x7fed48100fa0 msgr2=0x7fed481033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 -- 172.21.15.23:0/631456305 shutdown_connections 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.148+0000 7fed4fd50700 1 -- 172.21.15.23:0/631456305 wait complete. 2023-11-16T23:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.150+0000 7fed4fd50700 1 Processor -- start 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.150+0000 7fed4fd50700 1 -- start start 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.150+0000 7fed4fd50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4fd50700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4807c190 con 0x7fed48105910 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4daec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4daec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58802/0 (socket says 172.21.15.23:58802) 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4daec700 1 -- 172.21.15.23:0/2071769208 learned_addr learned my addr 172.21.15.23:0/2071769208 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4daec700 1 -- 172.21.15.23:0/2071769208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44009550 con 0x7fed48105910 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.151+0000 7fed4daec700 1 --2- 172.21.15.23:0/2071769208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed48106640 tx=0x7fed44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed44011630 con 0x7fed48105910 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44011790 con 0x7fed48105910 2023-11-16T23:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed48106e60 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed44011a30 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed48106cd0 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fed44022070 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.152+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed4401c450 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.153+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48062120 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.156+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fed4401b450 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.232+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fed481920f0 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.233+0000 7fed3effd700 1 -- 172.21.15.23:0/2071769208 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7fed4401b5f0 con 0x7fed48105910 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 msgr2=0x7fed4807bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 --2- 172.21.15.23:0/2071769208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed48106640 tx=0x7fed44004750 comp rx=0 tx=0).stop 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 shutdown_connections 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 --2- 172.21.15.23:0/2071769208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48105910 0x7fed4807bc50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 >> 172.21.15.23:0/2071769208 conn(0x7fed48100fa0 msgr2=0x7fed4818d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 shutdown_connections 2023-11-16T23:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:33.234+0000 7fed4fd50700 1 -- 172.21.15.23:0/2071769208 wait complete. 2023-11-16T23:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2023-11-16T23:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: fsmap 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: mgrmap e1: no daemons active 2023-11-16T23:50:33.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:33 smithi023 ceph-mon[97320]: from='client.? 172.21.15.23:0/2071769208' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T23:50:34.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:34 smithi023 ceph-mon[97320]: from='client.? 172.21.15.23:0/3065690157' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T23:50:34.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:34 smithi023 ceph-mon[97320]: from='client.? 172.21.15.23:0/3065690157' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T23:50:35.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:35.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T23:50:35.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:35.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T23:50:35.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.349+0000 7f14d61a5700 1 Processor -- start 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.349+0000 7f14d61a5700 1 -- start start 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.350+0000 7f14d61a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.350+0000 7f14d61a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d007a840 con 0x7f14d007be00 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.350+0000 7f14cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.350+0000 7f14cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58818/0 (socket says 172.21.15.23:58818) 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.350+0000 7f14cf7fe700 1 -- 172.21.15.23:0/3880279983 learned_addr learned my addr 172.21.15.23:0/3880279983 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.351+0000 7f14cf7fe700 1 -- 172.21.15.23:0/3880279983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d007a980 con 0x7f14d007be00 2023-11-16T23:50:35.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.351+0000 7f14cf7fe700 1 --2- 172.21.15.23:0/3880279983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f14c00098a0 tx=0x7f14c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bde3bec86e81892 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.351+0000 7f14ce7fc700 1 -- 172.21.15.23:0/3880279983 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14c0004160 con 0x7f14d007be00 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.351+0000 7f14ce7fc700 1 -- 172.21.15.23:0/3880279983 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c00042c0 con 0x7f14d007be00 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.351+0000 7f14ce7fc700 1 -- 172.21.15.23:0/3880279983 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14c00045a0 con 0x7f14d007be00 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 -- 172.21.15.23:0/3880279983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 msgr2=0x7f14d007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 --2- 172.21.15.23:0/3880279983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f14c00098a0 tx=0x7f14c0009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 -- 172.21.15.23:0/3880279983 shutdown_connections 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 --2- 172.21.15.23:0/3880279983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d007a300 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 -- 172.21.15.23:0/3880279983 >> 172.21.15.23:0/3880279983 conn(0x7f14d0100fa0 msgr2=0x7f14d01033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 -- 172.21.15.23:0/3880279983 shutdown_connections 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.352+0000 7f14d61a5700 1 -- 172.21.15.23:0/3880279983 wait complete. 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.353+0000 7f14d61a5700 1 Processor -- start 2023-11-16T23:50:35.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.353+0000 7f14d61a5700 1 -- start start 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.353+0000 7f14d61a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.353+0000 7f14d61a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d019b490 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.354+0000 7f14cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.354+0000 7f14cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58830/0 (socket says 172.21.15.23:58830) 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.354+0000 7f14cf7fe700 1 -- 172.21.15.23:0/3065690157 learned_addr learned my addr 172.21.15.23:0/3065690157 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.354+0000 7f14cf7fe700 1 -- 172.21.15.23:0/3065690157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0009550 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.354+0000 7f14cf7fe700 1 --2- 172.21.15.23:0/3065690157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f14c0004000 tx=0x7f14c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14c0011630 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c0011790 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14c0011a30 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d007ae80 con 0x7f14d007be00 2023-11-16T23:50:35.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d007acf0 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f14c0022070 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.355+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f14c001c450 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.356+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d0194e80 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.358+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f14c001b450 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.431+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f14d0062120 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.436+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14c001b5f0 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.436+0000 7f14ccff9700 1 -- 172.21.15.23:0/3065690157 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f14d0062120 con 0x7f14d007be00 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 msgr2=0x7f14d019af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 --2- 172.21.15.23:0/3065690157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f14c0004000 tx=0x7f14c0004750 comp rx=0 tx=0).stop 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 shutdown_connections 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 --2- 172.21.15.23:0/3065690157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14d007be00 0x7f14d019af50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 >> 172.21.15.23:0/3065690157 conn(0x7f14d0100fa0 msgr2=0x7f14d01027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:35.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 shutdown_connections 2023-11-16T23:50:35.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:34.437+0000 7f14d61a5700 1 -- 172.21.15.23:0/3065690157 wait complete. 2023-11-16T23:50:35.026 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2023-11-16T23:50:35.888 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:35 smithi023 ceph-mon[97320]: from='client.? 172.21.15.23:0/3148984786' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:50:36.201 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.524+0000 7f3c3bea4700 1 Processor -- start 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.524+0000 7f3c3bea4700 1 -- start start 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.524+0000 7f3c3bea4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.524+0000 7f3c3bea4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3407a820 con 0x7f3c3407bde0 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.525+0000 7f3c39c40700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.525+0000 7f3c39c40700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58842/0 (socket says 172.21.15.23:58842) 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.525+0000 7f3c39c40700 1 -- 172.21.15.23:0/2378463461 learned_addr learned my addr 172.21.15.23:0/2378463461 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.525+0000 7f3c39c40700 1 -- 172.21.15.23:0/2378463461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3407a960 con 0x7f3c3407bde0 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c39c40700 1 --2- 172.21.15.23:0/2378463461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3c300098a0 tx=0x7f3c30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bd306690772439 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c38c3e700 1 -- 172.21.15.23:0/2378463461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c30004160 con 0x7f3c3407bde0 2023-11-16T23:50:36.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c38c3e700 1 -- 172.21.15.23:0/2378463461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c300042c0 con 0x7f3c3407bde0 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c38c3e700 1 -- 172.21.15.23:0/2378463461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c30011a00 con 0x7f3c3407bde0 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c3bea4700 1 -- 172.21.15.23:0/2378463461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 msgr2=0x7f3c3407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.526+0000 7f3c3bea4700 1 --2- 172.21.15.23:0/2378463461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3c300098a0 tx=0x7f3c30009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.527+0000 7f3c3bea4700 1 -- 172.21.15.23:0/2378463461 shutdown_connections 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.527+0000 7f3c3bea4700 1 --2- 172.21.15.23:0/2378463461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3407a2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.527+0000 7f3c3bea4700 1 -- 172.21.15.23:0/2378463461 >> 172.21.15.23:0/2378463461 conn(0x7f3c34100fa0 msgr2=0x7f3c341033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.527+0000 7f3c3bea4700 1 -- 172.21.15.23:0/2378463461 shutdown_connections 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.527+0000 7f3c3bea4700 1 -- 172.21.15.23:0/2378463461 wait complete. 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c3bea4700 1 Processor -- start 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c3bea4700 1 -- start start 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c3bea4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c3bea4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3419bcf0 con 0x7f3c3407bde0 2023-11-16T23:50:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c39c40700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c39c40700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58850/0 (socket says 172.21.15.23:58850) 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.528+0000 7f3c39c40700 1 -- 172.21.15.23:0/3148984786 learned_addr learned my addr 172.21.15.23:0/3148984786 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c39c40700 1 -- 172.21.15.23:0/3148984786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30009550 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c39c40700 1 --2- 172.21.15.23:0/3148984786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c30006af0 tx=0x7f3c30011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c30003a40 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c30003ba0 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c3001a700 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3410be00 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.529+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3410bc70 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.530+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3c3001a930 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.530+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3c3002dd10 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.531+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c34062120 con 0x7f3c3407bde0 2023-11-16T23:50:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.533+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f3c300248d0 con 0x7f3c3407bde0 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.608+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3c3419c230 con 0x7f3c3407bde0 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.608+0000 7f3c2affd700 1 -- 172.21.15.23:0/3148984786 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3c3001ad20 con 0x7f3c3407bde0 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 msgr2=0x7f3c3419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 --2- 172.21.15.23:0/3148984786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c30006af0 tx=0x7f3c30011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 shutdown_connections 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 --2- 172.21.15.23:0/3148984786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c3407bde0 0x7f3c3419b7b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 >> 172.21.15.23:0/3148984786 conn(0x7f3c34100fa0 msgr2=0x7f3c34101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 shutdown_connections 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:35.609+0000 7f3c3bea4700 1 -- 172.21.15.23:0/3148984786 wait complete. 2023-11-16T23:50:36.205 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2023-11-16T23:50:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:36 smithi023 systemd[1]: Stopping Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:50:37.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a[97296]: 2023-11-16T23:50:36.658+0000 7fa1f12cd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T23:50:37.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a[97296]: 2023-11-16T23:50:36.658+0000 7fa1f12cd700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T23:50:37.364 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:37 smithi023 bash[97656]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T23:50:37.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:37 smithi023 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service: Succeeded. 2023-11-16T23:50:37.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:37 smithi023 systemd[1]: Stopped Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:50:37.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:37 smithi023 systemd[1]: Starting Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:50:38.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 podman[97784]: 2023-11-16T23:50:38.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 bash[97784]: e083aa47914fbf96a1f882fde8d86d84ae4d3a3761f4cfb1fb3cf59d0f09f378 2023-11-16T23:50:38.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T23:50:38.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev), process ceph-mon, pid 7 2023-11-16T23:50:38.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: pidfile_write: ignore empty --pid-file 2023-11-16T23:50:38.337 INFO:teuthology.orchestra.run.smithi023.stdout:Setting mon public_network to 172.21.0.0/20 2023-11-16T23:50:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: load: jerasure load: lrc 2023-11-16T23:50:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: RocksDB version: 7.9.2 2023-11-16T23:50:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Git sha 0 2023-11-16T23:50:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Compile date 2023-11-16 19:40:53 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: DB SUMMARY 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: DB Session ID: 8VY9SMIJJR552Q2FRCCV 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: CURRENT file: CURRENT 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97726 ; 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.error_if_exists: 0 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.create_if_missing: 0 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.paranoid_checks: 1 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T23:50:38.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.env: 0x55fa32444720 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.info_log: 0x55fa3581f3c0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.statistics: (nil) 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.use_fsync: 0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_log_file_size: 0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_fallocate: 1 2023-11-16T23:50:38.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.use_direct_reads: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.db_log_dir: 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.wal_dir: 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T23:50:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.write_buffer_manager: 0x55fa34a903c0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.unordered_write: 0 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T23:50:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.row_cache: None 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.wal_filter: None 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.two_write_queues: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.wal_compression: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.atomic_flush: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T23:50:38.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.log_readahead_size: 0 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_background_jobs: 2 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_background_compactions: -1 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_subcompactions: 1 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T23:50:38.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_open_files: -1 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T23:50:38.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_background_flushes: -1 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Compression algorithms supported: 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kZSTD supported: 0 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kXpressCompression supported: 0 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kZlibCompression supported: 1 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kSnappyCompression supported: 1 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kLZ4Compression supported: 1 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: kBZip2Compression supported: 0 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-11-16T23:50:38.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.merge_operator: 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_filter: None 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa3581f4e0) 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2023-11-16T23:50:38.527 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x55fa34b311f0 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2023-11-16T23:50:38.528 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2023-11-16T23:50:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression: NoCompression 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.num_levels: 7 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T23:50:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T23:50:38.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:50:38.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T23:50:38.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T23:50:38.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.table_properties_collectors: 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.inplace_update_support: 0 2023-11-16T23:50:38.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.bloom_locality: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.max_successive_merges: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.ttl: 2592000 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T23:50:38.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enable_blob_files: false 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.min_blob_size: 0 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 860ca195-1414-40f1-88e1-f1227ed45230 2023-11-16T23:50:38.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178638275733, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178638277652, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 91056, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13416, "raw_average_key_size": 49, "raw_value_size": 83748, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700178638, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "860ca195-1414-40f1-88e1-f1227ed45230", "db_session_id": "8VY9SMIJJR552Q2FRCCV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178638277793, "job": 1, "event": "recovery_finished"} 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa34bce000 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: DB pointer 0x55fa34bba000 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:50:38.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 92.69 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 92.69 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 68.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T23:50:38.540 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x55fa34b311f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???) e1 preinit fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).mds e1 new map 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).mds e1 print_map 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T23:50:38.541 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expand map: {default=false} 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta from 'false' to 'false' 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expanded map: {default=false} 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expand map: {default=info} 2023-11-16T23:50:38.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta from 'info' to 'info' 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expanded map: {default=info} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expand map: {default=daemon} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expand map: {default=debug} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: expand_channel_meta expanded map: {default=debug} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 systemd[1]: Started Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: fsmap 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T23:50:38.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:38 smithi023 ceph-mon[97824]: mgrmap e1: no daemons active 2023-11-16T23:50:39.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.895+0000 7f625fefc700 1 Processor -- start 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.895+0000 7f625fefc700 1 -- start start 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.896+0000 7f625fefc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.896+0000 7f625fefc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258106260 con 0x7f6258105910 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.896+0000 7f625dc98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.896+0000 7f625dc98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58858/0 (socket says 172.21.15.23:58858) 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.896+0000 7f625dc98700 1 -- 172.21.15.23:0/3929652096 learned_addr learned my addr 172.21.15.23:0/3929652096 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:39.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.897+0000 7f625dc98700 1 -- 172.21.15.23:0/3929652096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62581063a0 con 0x7f6258105910 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.897+0000 7f625dc98700 1 --2- 172.21.15.23:0/3929652096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f62540098a0 tx=0x7f6254009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51cd02fb8dfd6e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.897+0000 7f625cc96700 1 -- 172.21.15.23:0/3929652096 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6254004160 con 0x7f6258105910 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.897+0000 7f625cc96700 1 -- 172.21.15.23:0/3929652096 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62540042c0 con 0x7f6258105910 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.897+0000 7f625cc96700 1 -- 172.21.15.23:0/3929652096 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6254011a00 con 0x7f6258105910 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 -- 172.21.15.23:0/3929652096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 msgr2=0x7f6258105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 --2- 172.21.15.23:0/3929652096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f62540098a0 tx=0x7f6254009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 -- 172.21.15.23:0/3929652096 shutdown_connections 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 --2- 172.21.15.23:0/3929652096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258105d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 -- 172.21.15.23:0/3929652096 >> 172.21.15.23:0/3929652096 conn(0x7f6258100fa0 msgr2=0x7f62581033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 -- 172.21.15.23:0/3929652096 shutdown_connections 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.898+0000 7f625fefc700 1 -- 172.21.15.23:0/3929652096 wait complete. 2023-11-16T23:50:39.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.899+0000 7f625fefc700 1 Processor -- start 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625fefc700 1 -- start start 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625fefc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625fefc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258199a90 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625dc98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625dc98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58862/0 (socket says 172.21.15.23:58862) 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625dc98700 1 -- 172.21.15.23:0/3819058753 learned_addr learned my addr 172.21.15.23:0/3819058753 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.900+0000 7f625dc98700 1 -- 172.21.15.23:0/3819058753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6254009550 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f625dc98700 1 --2- 172.21.15.23:0/3819058753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6254006af0 tx=0x7f6254011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6254003a40 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6254003ba0 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f625401a6e0 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258107960 con 0x7f6258105910 2023-11-16T23:50:39.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62581077d0 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.901+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f625401ab60 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.902+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6254019700 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.902+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6258109de0 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.905+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f625402dde0 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.977+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6258062120 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.980+0000 7f624effd700 1 -- 172.21.15.23:0/3819058753 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6254024a60 con 0x7f6258105910 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 msgr2=0x7f6258199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 --2- 172.21.15.23:0/3819058753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6254006af0 tx=0x7f6254011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 shutdown_connections 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 --2- 172.21.15.23:0/3819058753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6258105910 0x7f6258199550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 >> 172.21.15.23:0/3819058753 conn(0x7f6258100fa0 msgr2=0x7f6258101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 shutdown_connections 2023-11-16T23:50:39.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:38.981+0000 7f625fefc700 1 -- 172.21.15.23:0/3819058753 wait complete. 2023-11-16T23:50:39.429 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2023-11-16T23:50:39.429 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T23:50:39.429 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2023-11-16T23:50:39.429 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 9283 ... 2023-11-16T23:50:39.429 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 8765 ... 2023-11-16T23:50:39.723 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T23:50:39.723 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x.service: Unit ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x.service not loaded. 2023-11-16T23:50:39.938 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7.target.wants/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T23:50:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3819058753' entity='client.admin' 2023-11-16T23:50:40.018 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:39 smithi023 systemd[1]: Starting Ceph mgr.x for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:50:40.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:40 smithi023 podman[98056]: 2023-11-16T23:50:40.866 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T23:50:40.866 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2023-11-16T23:50:40.882 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T23:50:40.882 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:50:40.882 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2023-11-16T23:50:40.883 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T23:50:40.902 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T23:50:40.902 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2023-11-16T23:50:40.917 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T23:50:40.917 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:50:40.917 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2023-11-16T23:50:40.918 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-11-16T23:50:40.918 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2023-11-16T23:50:40.918 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2023-11-16T23:50:41.113 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:40 smithi023 bash[98056]: fb5144615a3bf62b87b7b7c9c3a5ba2e0c45e738b877b7352be6cf50de0eced5 2023-11-16T23:50:41.113 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:40 smithi023 systemd[1]: Started Ceph mgr.x for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:50:41.113 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:40 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:40.881+0000 7f7629379200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T23:50:41.114 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:40 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:40.993+0000 7f7629379200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T23:50:41.114 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:41.111+0000 7f7629379200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T23:50:41.478 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:41.239+0000 7f7629379200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T23:50:41.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:41 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1888613158' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T23:50:41.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:41.476+0000 7f7629379200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T23:50:42.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:41.908+0000 7f7629379200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T23:50:42.324 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:42.027+0000 7f7629379200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T23:50:42.324 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:42.194+0000 7f7629379200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T23:50:42.324 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:42.322+0000 7f7629379200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T23:50:43.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:42.889+0000 7f7629379200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T23:50:44.017 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:43.710+0000 7f7629379200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T23:50:44.018 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:43.924+0000 7f7629379200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T23:50:45.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:44.913+0000 7f7629379200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:50:45.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:45.037+0000 7f7629379200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T23:50:45.645 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:45.268+0000 7f7629379200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:50:45.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:45.822 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:50:45.822 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T23:50:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T23:50:45.824 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T23:50:45.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T23:50:45.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T23:50:32.514980+0000", 2023-11-16T23:50:45.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.485+0000 7fda87402700 1 Processor -- start 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.485+0000 7fda87402700 1 -- start start 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.486+0000 7fda87402700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.486+0000 7fda87402700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80106260 con 0x7fda80105910 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.486+0000 7fda8519e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.486+0000 7fda8519e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55060/0 (socket says 172.21.15.23:55060) 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.486+0000 7fda8519e700 1 -- 172.21.15.23:0/1305225811 learned_addr learned my addr 172.21.15.23:0/1305225811 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.487+0000 7fda8519e700 1 -- 172.21.15.23:0/1305225811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda801063a0 con 0x7fda80105910 2023-11-16T23:50:45.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.487+0000 7fda8519e700 1 --2- 172.21.15.23:0/1305225811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098a0 tx=0x7fda7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5dfc8449648648 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.487+0000 7fda77fff700 1 -- 172.21.15.23:0/1305225811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c004160 con 0x7fda80105910 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.487+0000 7fda77fff700 1 -- 172.21.15.23:0/1305225811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda7c0042c0 con 0x7fda80105910 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.487+0000 7fda77fff700 1 -- 172.21.15.23:0/1305225811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c011a00 con 0x7fda80105910 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 -- 172.21.15.23:0/1305225811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 msgr2=0x7fda80105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 --2- 172.21.15.23:0/1305225811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098a0 tx=0x7fda7c009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 -- 172.21.15.23:0/1305225811 shutdown_connections 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 --2- 172.21.15.23:0/1305225811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda80105d20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 -- 172.21.15.23:0/1305225811 >> 172.21.15.23:0/1305225811 conn(0x7fda80100fa0 msgr2=0x7fda801033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 -- 172.21.15.23:0/1305225811 shutdown_connections 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.488+0000 7fda87402700 1 -- 172.21.15.23:0/1305225811 wait complete. 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.489+0000 7fda87402700 1 Processor -- start 2023-11-16T23:50:45.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.489+0000 7fda87402700 1 -- start start 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda87402700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda87402700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8007c190 con 0x7fda80105910 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda8519e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda8519e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55062/0 (socket says 172.21.15.23:55062) 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda8519e700 1 -- 172.21.15.23:0/1888613158 learned_addr learned my addr 172.21.15.23:0/1888613158 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.490+0000 7fda8519e700 1 -- 172.21.15.23:0/1888613158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c009550 con 0x7fda80105910 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda8519e700 1 --2- 172.21.15.23:0/1888613158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda7c006af0 tx=0x7fda7c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c003a40 con 0x7fda80105910 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda7c003ba0 con 0x7fda80105910 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c01a6e0 con 0x7fda80105910 2023-11-16T23:50:45.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80107960 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.491+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda801077d0 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.492+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda7c01ab60 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.492+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fda7c019700 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.492+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda80062120 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.497+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fda7c02dde0 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.574+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fda80109db0 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.575+0000 7fda767fc700 1 -- 172.21.15.23:0/1888613158 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fda7c024a60 con 0x7fda80105910 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.575+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 msgr2=0x7fda8007bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 --2- 172.21.15.23:0/1888613158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda7c006af0 tx=0x7fda7c011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:45.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 shutdown_connections 2023-11-16T23:50:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 --2- 172.21.15.23:0/1888613158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda80105910 0x7fda8007bc50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 >> 172.21.15.23:0/1888613158 conn(0x7fda80100fa0 msgr2=0x7fda80101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 shutdown_connections 2023-11-16T23:50:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:41.576+0000 7fda87402700 1 -- 172.21.15.23:0/1888613158 wait complete. 2023-11-16T23:50:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2023-11-16T23:50:45.944 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:45.644+0000 7f7629379200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T23:50:46.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:45.934+0000 7f7629379200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T23:50:46.268 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:46.135+0000 7f7629379200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T23:50:46.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:46.491+0000 7f7629379200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T23:50:47.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:47.464+0000 7f7629379200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T23:50:47.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:47.644+0000 7f7629379200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T23:50:48.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:47.905+0000 7f7629379200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T23:50:48.268 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:48.060+0000 7f7629379200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T23:50:48.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:48.365+0000 7f7629379200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T23:50:49.727 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:49.513+0000 7f7629379200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: Activating manager daemon x 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: mgrmap e2: x(active, starting, since 0.00593857s) 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T23:50:49.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: Manager daemon x is now available 2023-11-16T23:50:49.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T23:50:49.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T23:50:49.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' 2023-11-16T23:50:49.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' 2023-11-16T23:50:49.729 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:49 smithi023 ceph-mon[97824]: from='mgr.14100 172.21.15.23:0/3676893245' entity='mgr.x' 2023-11-16T23:50:50.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T23:50:50.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:50.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T23:50:50.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T23:50:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T23:50:32.514980+0000", 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:50.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.617+0000 7f6dccd5c700 1 Processor -- start 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.618+0000 7f6dccd5c700 1 -- start start 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.618+0000 7f6dccd5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:50.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.618+0000 7f6dccd5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8105140 con 0x7f6dc81047f0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.619+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.619+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55158/0 (socket says 172.21.15.23:55158) 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.619+0000 7f6dc659c700 1 -- 172.21.15.23:0/4122599247 learned_addr learned my addr 172.21.15.23:0/4122599247 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.619+0000 7f6dc659c700 1 -- 172.21.15.23:0/4122599247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8105280 con 0x7f6dc81047f0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.619+0000 7f6dc659c700 1 --2- 172.21.15.23:0/4122599247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6db800bb40 tx=0x7f6db800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32f5f6589864f6c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dc559a700 1 -- 172.21.15.23:0/4122599247 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6db800dbe0 con 0x7f6dc81047f0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dc559a700 1 -- 172.21.15.23:0/4122599247 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6db80044d0 con 0x7f6dc81047f0 2023-11-16T23:50:50.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dc559a700 1 -- 172.21.15.23:0/4122599247 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6db8010470 con 0x7f6dc81047f0 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dccd5c700 1 -- 172.21.15.23:0/4122599247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 msgr2=0x7f6dc8104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dccd5c700 1 --2- 172.21.15.23:0/4122599247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6db800bb40 tx=0x7f6db800be50 comp rx=0 tx=0).stop 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dccd5c700 1 -- 172.21.15.23:0/4122599247 shutdown_connections 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dccd5c700 1 --2- 172.21.15.23:0/4122599247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc8104c00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.620+0000 7f6dccd5c700 1 -- 172.21.15.23:0/4122599247 >> 172.21.15.23:0/4122599247 conn(0x7f6dc80ffe40 msgr2=0x7f6dc8102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:50.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.621+0000 7f6dccd5c700 1 -- 172.21.15.23:0/4122599247 shutdown_connections 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.621+0000 7f6dccd5c700 1 -- 172.21.15.23:0/4122599247 wait complete. 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dccd5c700 1 Processor -- start 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dccd5c700 1 -- start start 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dccd5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dccd5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc819bc90 con 0x7f6dc81047f0 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dc659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55162/0 (socket says 172.21.15.23:55162) 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.622+0000 7f6dc659c700 1 -- 172.21.15.23:0/93674112 learned_addr learned my addr 172.21.15.23:0/93674112 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6dc659c700 1 -- 172.21.15.23:0/93674112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db800b7f0 con 0x7f6dc81047f0 2023-11-16T23:50:50.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6dc659c700 1 --2- 172.21.15.23:0/93674112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6db800b340 tx=0x7f6db8003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6db8004340 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6db8010bd0 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc8106840 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6db8021c60 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.623+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc81066b0 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.624+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc8195530 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.625+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f6db8021410 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.625+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6db804c370 con 0x7f6dc81047f0 2023-11-16T23:50:50.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.627+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f6db800f5b0 con 0x7f6dc81047f0 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.704+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6dc802cfe0 con 0x7f6dc81047f0 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.705+0000 7f6db77fe700 1 -- 172.21.15.23:0/93674112 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f6db800f3d0 con 0x7f6dc81047f0 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 msgr2=0x7f6dc819b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 --2- 172.21.15.23:0/93674112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6db800b340 tx=0x7f6db8003df0 comp rx=0 tx=0).stop 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 shutdown_connections 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 --2- 172.21.15.23:0/93674112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6dc81047f0 0x7f6dc819b750 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 >> 172.21.15.23:0/93674112 conn(0x7f6dc80ffe40 msgr2=0x7f6dc8192670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 shutdown_connections 2023-11-16T23:50:50.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:49.706+0000 7f6dccd5c700 1 -- 172.21.15.23:0/93674112 wait complete. 2023-11-16T23:50:50.296 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2023-11-16T23:50:51.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:50 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/93674112' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T23:50:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:50 smithi023 ceph-mon[97824]: mgrmap e3: x(active, since 1.00882s) 2023-11-16T23:50:52.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:52 smithi023 ceph-mon[97824]: mgrmap e4: x(active, since 2s) 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T23:50:53.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T23:50:53.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T23:50:53.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T23:50:53.423 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T23:50:32.514980+0000", 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T23:50:53.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2023-11-16T23:50:53.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T23:50:53.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:50:53.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2213e41700 1 Processor -- start 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2213e41700 1 -- start start 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2213e41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2213e41700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c106260 con 0x7f220c105910 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2211bdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2211bdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51094/0 (socket says 172.21.15.23:51094) 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.679+0000 7f2211bdd700 1 -- 172.21.15.23:0/75689240 learned_addr learned my addr 172.21.15.23:0/75689240 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.680+0000 7f2211bdd700 1 -- 172.21.15.23:0/75689240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c1063a0 con 0x7f220c105910 2023-11-16T23:50:53.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.680+0000 7f2211bdd700 1 --2- 172.21.15.23:0/75689240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f22080098a0 tx=0x7f2208009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c742c729453ae9bb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:53.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.680+0000 7f2210bdb700 1 -- 172.21.15.23:0/75689240 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2208004160 con 0x7f220c105910 2023-11-16T23:50:53.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.680+0000 7f2210bdb700 1 -- 172.21.15.23:0/75689240 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22080042c0 con 0x7f220c105910 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.680+0000 7f2210bdb700 1 -- 172.21.15.23:0/75689240 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2208011a00 con 0x7f220c105910 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 -- 172.21.15.23:0/75689240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 msgr2=0x7f220c105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 --2- 172.21.15.23:0/75689240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f22080098a0 tx=0x7f2208009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 -- 172.21.15.23:0/75689240 shutdown_connections 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 --2- 172.21.15.23:0/75689240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c105d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 -- 172.21.15.23:0/75689240 >> 172.21.15.23:0/75689240 conn(0x7f220c100fa0 msgr2=0x7f220c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 -- 172.21.15.23:0/75689240 shutdown_connections 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.681+0000 7f2213e41700 1 -- 172.21.15.23:0/75689240 wait complete. 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.682+0000 7f2213e41700 1 Processor -- start 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.682+0000 7f2213e41700 1 -- start start 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2213e41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2213e41700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c07c190 con 0x7f220c105910 2023-11-16T23:50:53.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2211bdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2211bdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51108/0 (socket says 172.21.15.23:51108) 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2211bdd700 1 -- 172.21.15.23:0/3766746472 learned_addr learned my addr 172.21.15.23:0/3766746472 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2211bdd700 1 -- 172.21.15.23:0/3766746472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2208009550 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.683+0000 7f2211bdd700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2208006af0 tx=0x7f2208011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.684+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2208003a40 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.684+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2208003ba0 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.684+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c107960 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.684+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f220801a6e0 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.684+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c1077d0 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.685+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f2208024a60 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.685+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c062120 con 0x7f220c105910 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.685+0000 7f2202ffd700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 0x7f21f803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:53.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.685+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f220804c690 con 0x7f220c105910 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.685+0000 7f22113dc700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 0x7f21f803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.686+0000 7f22113dc700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 0x7f21f803ac80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21fc006fd0 tx=0x7f21fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.691+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2208024430 con 0x7f220c105910 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.977+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f220c07b170 con 0x7f220c105910 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.977+0000 7f2202ffd700 1 -- 172.21.15.23:0/3766746472 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f220c07b170 con 0x7f220c105910 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 msgr2=0x7f21f803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 0x7f21f803ac80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f21fc006fd0 tx=0x7f21fc006e40 comp rx=0 tx=0).stop 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 msgr2=0x7f220c07bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2208006af0 tx=0x7f2208011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 shutdown_connections 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f21f80387d0 0x7f21f803ac80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 --2- 172.21.15.23:0/3766746472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f220c105910 0x7f220c07bc50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:53.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 >> 172.21.15.23:0/3766746472 conn(0x7f220c100fa0 msgr2=0x7f220c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:53.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.979+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 shutdown_connections 2023-11-16T23:50:53.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:52.980+0000 7f2213e41700 1 -- 172.21.15.23:0/3766746472 wait complete. 2023-11-16T23:50:53.431 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2023-11-16T23:50:53.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:53 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3766746472' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T23:50:54.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.938+0000 7f4b3b20d700 1 Processor -- start 2023-11-16T23:50:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.938+0000 7f4b3b20d700 1 -- start start 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.938+0000 7f4b3b20d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.938+0000 7f4b3b20d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3407a860 con 0x7f4b3407be20 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.939+0000 7f4b38fa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.939+0000 7f4b38fa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51118/0 (socket says 172.21.15.23:51118) 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.939+0000 7f4b38fa9700 1 -- 172.21.15.23:0/3893704084 learned_addr learned my addr 172.21.15.23:0/3893704084 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.939+0000 7f4b38fa9700 1 -- 172.21.15.23:0/3893704084 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3407a9a0 con 0x7f4b3407be20 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.939+0000 7f4b38fa9700 1 --2- 172.21.15.23:0/3893704084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b280098a0 tx=0x7f4b28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ec229295f6b4da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b337fe700 1 -- 172.21.15.23:0/3893704084 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b28004160 con 0x7f4b3407be20 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b337fe700 1 -- 172.21.15.23:0/3893704084 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b280042c0 con 0x7f4b3407be20 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b337fe700 1 -- 172.21.15.23:0/3893704084 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b28011a00 con 0x7f4b3407be20 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 -- 172.21.15.23:0/3893704084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 msgr2=0x7f4b3407a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/3893704084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b280098a0 tx=0x7f4b28009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:54.600 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 -- 172.21.15.23:0/3893704084 shutdown_connections 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/3893704084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3407a320 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 -- 172.21.15.23:0/3893704084 >> 172.21.15.23:0/3893704084 conn(0x7f4b34100fa0 msgr2=0x7f4b341033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 -- 172.21.15.23:0/3893704084 shutdown_connections 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.940+0000 7f4b3b20d700 1 -- 172.21.15.23:0/3893704084 wait complete. 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.941+0000 7f4b3b20d700 1 Processor -- start 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b3b20d700 1 -- start start 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b3b20d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b3b20d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3419bce0 con 0x7f4b3407be20 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b38fa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b38fa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51128/0 (socket says 172.21.15.23:51128) 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b38fa9700 1 -- 172.21.15.23:0/884175022 learned_addr learned my addr 172.21.15.23:0/884175022 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.942+0000 7f4b38fa9700 1 -- 172.21.15.23:0/884175022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009550 con 0x7f4b3407be20 2023-11-16T23:50:54.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b38fa9700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009fb0 tx=0x7f4b28011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b28003930 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b28003a90 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3410bdc0 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b280195e0 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.943+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3410bc30 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.944+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b34062120 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.944+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f4b2801a9d0 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.945+0000 7f4b31ffb700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 0x7f4b2403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.945+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b2804c3d0 con 0x7f4b3407be20 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.945+0000 7f4b33fff700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 0x7f4b2403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.946+0000 7f4b33fff700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 0x7f4b2403ac30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b20006fd0 tx=0x7f4b20006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:54.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:53.950+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4b28003c00 con 0x7f4b3407be20 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.155+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4b3419c470 con 0x7f4b3407be20 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.156+0000 7f4b31ffb700 1 -- 172.21.15.23:0/884175022 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f4b28023b40 con 0x7f4b3407be20 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 msgr2=0x7f4b2403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 0x7f4b2403ac30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b20006fd0 tx=0x7f4b20006e40 comp rx=0 tx=0).stop 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 msgr2=0x7f4b3419b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009fb0 tx=0x7f4b28011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 shutdown_connections 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f4b24038780 0x7f4b2403ac30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 --2- 172.21.15.23:0/884175022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b3407be20 0x7f4b3419b7a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 >> 172.21.15.23:0/884175022 conn(0x7f4b34100fa0 msgr2=0x7f4b34101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 shutdown_connections 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:54.159+0000 7f4b3b20d700 1 -- 172.21.15.23:0/884175022 wait complete. 2023-11-16T23:50:54.603 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2023-11-16T23:50:54.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:54 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/884175022' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T23:50:55.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:55 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2172821472' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T23:50:55.876 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: ignoring --setuser ceph since I am not root 2023-11-16T23:50:55.876 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: ignoring --setgroup ceph since I am not root 2023-11-16T23:50:55.876 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:55.873+0000 7f2a062b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T23:50:56.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.090+0000 7f4353edd700 1 Processor -- start 2023-11-16T23:50:56.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.090+0000 7f4353edd700 1 -- start start 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.090+0000 7f4353edd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.090+0000 7f4353edd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c106260 con 0x7f434c105910 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.091+0000 7f4351c79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.091+0000 7f4351c79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51134/0 (socket says 172.21.15.23:51134) 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.091+0000 7f4351c79700 1 -- 172.21.15.23:0/732737410 learned_addr learned my addr 172.21.15.23:0/732737410 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.091+0000 7f4351c79700 1 -- 172.21.15.23:0/732737410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c1063a0 con 0x7f434c105910 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.091+0000 7f4351c79700 1 --2- 172.21.15.23:0/732737410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43480098a0 tx=0x7f4348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b878be411cbc8d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4350c77700 1 -- 172.21.15.23:0/732737410 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4348004160 con 0x7f434c105910 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4350c77700 1 -- 172.21.15.23:0/732737410 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43480042c0 con 0x7f434c105910 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4350c77700 1 -- 172.21.15.23:0/732737410 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4348011a00 con 0x7f434c105910 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 -- 172.21.15.23:0/732737410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 msgr2=0x7f434c105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 --2- 172.21.15.23:0/732737410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43480098a0 tx=0x7f4348009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:56.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 -- 172.21.15.23:0/732737410 shutdown_connections 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 --2- 172.21.15.23:0/732737410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c105d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 -- 172.21.15.23:0/732737410 >> 172.21.15.23:0/732737410 conn(0x7f434c100fa0 msgr2=0x7f434c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 -- 172.21.15.23:0/732737410 shutdown_connections 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.092+0000 7f4353edd700 1 -- 172.21.15.23:0/732737410 wait complete. 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4353edd700 1 Processor -- start 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4353edd700 1 -- start start 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4353edd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4353edd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c07c190 con 0x7f434c105910 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4351c79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4351c79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51138/0 (socket says 172.21.15.23:51138) 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.094+0000 7f4351c79700 1 -- 172.21.15.23:0/2172821472 learned_addr learned my addr 172.21.15.23:0/2172821472 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4351c79700 1 -- 172.21.15.23:0/2172821472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4348009550 con 0x7f434c105910 2023-11-16T23:50:56.088 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4351c79700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4348006af0 tx=0x7f4348011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4348003a40 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4348003ba0 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f434801a6e0 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c107960 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.095+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c1077d0 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.096+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f4348024a60 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.096+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c062120 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.097+0000 7f4342ffd700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 0x7f433803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.097+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f434804c690 con 0x7f434c105910 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.097+0000 7f4351478700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 0x7f433803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.098+0000 7f4351478700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 0x7f433803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f433c006fd0 tx=0x7f433c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:56.089 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.102+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4348024430 con 0x7f434c105910 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.367+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f434c07b170 con 0x7f434c105910 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.596+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f434802d910 con 0x7f434c105910 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.598+0000 7f4342ffd700 1 -- 172.21.15.23:0/2172821472 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f434802c350 con 0x7f434c105910 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.600+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 msgr2=0x7f433803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 0x7f433803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f433c006fd0 tx=0x7f433c006e40 comp rx=0 tx=0).stop 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 msgr2=0x7f434c07bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4348006af0 tx=0x7f4348011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 shutdown_connections 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f43380387d0 0x7f433803ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 --2- 172.21.15.23:0/2172821472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f434c105910 0x7f434c07bc50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 >> 172.21.15.23:0/2172821472 conn(0x7f434c100fa0 msgr2=0x7f434c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:56.090 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 shutdown_connections 2023-11-16T23:50:56.091 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:55.601+0000 7f4353edd700 1 -- 172.21.15.23:0/2172821472 wait complete. 2023-11-16T23:50:56.229 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:55.981+0000 7f2a062b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T23:50:56.229 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:56.099+0000 7f2a062b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T23:50:56.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:56.228+0000 7f2a062b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T23:50:56.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:56.465+0000 7f2a062b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T23:50:56.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2172821472' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T23:50:56.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-mon[97824]: mgrmap e5: x(active, since 6s) 2023-11-16T23:50:57.183 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:56.873+0000 7f2a062b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T23:50:57.183 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:57.009+0000 7f2a062b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.616+0000 7f7415ae7700 1 Processor -- start 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.616+0000 7f7415ae7700 1 -- start start 2023-11-16T23:50:57.269 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.616+0000 7f7415ae7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.616+0000 7f7415ae7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741007a840 con 0x7f741007be00 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.617+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.617+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51174/0 (socket says 172.21.15.23:51174) 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.617+0000 7f740f7fe700 1 -- 172.21.15.23:0/519110522 learned_addr learned my addr 172.21.15.23:0/519110522 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.617+0000 7f740f7fe700 1 -- 172.21.15.23:0/519110522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741007a980 con 0x7f741007be00 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.617+0000 7f740f7fe700 1 --2- 172.21.15.23:0/519110522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f74000098a0 tx=0x7f7400009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c52e6e41c201772 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.618+0000 7f740e7fc700 1 -- 172.21.15.23:0/519110522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7400004160 con 0x7f741007be00 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.618+0000 7f740e7fc700 1 -- 172.21.15.23:0/519110522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74000042c0 con 0x7f741007be00 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.618+0000 7f740e7fc700 1 -- 172.21.15.23:0/519110522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7400011a00 con 0x7f741007be00 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.618+0000 7f7415ae7700 1 -- 172.21.15.23:0/519110522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 msgr2=0x7f741007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.618+0000 7f7415ae7700 1 --2- 172.21.15.23:0/519110522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f74000098a0 tx=0x7f7400009bb0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.270 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.619+0000 7f7415ae7700 1 -- 172.21.15.23:0/519110522 shutdown_connections 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.619+0000 7f7415ae7700 1 --2- 172.21.15.23:0/519110522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741007a300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.619+0000 7f7415ae7700 1 -- 172.21.15.23:0/519110522 >> 172.21.15.23:0/519110522 conn(0x7f7410100fa0 msgr2=0x7f74101033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.619+0000 7f7415ae7700 1 -- 172.21.15.23:0/519110522 shutdown_connections 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.619+0000 7f7415ae7700 1 -- 172.21.15.23:0/519110522 wait complete. 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.620+0000 7f7415ae7700 1 Processor -- start 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.620+0000 7f7415ae7700 1 -- start start 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.620+0000 7f7415ae7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.620+0000 7f7415ae7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741019bd00 con 0x7f741007be00 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51176/0 (socket says 172.21.15.23:51176) 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740f7fe700 1 -- 172.21.15.23:0/3278078143 learned_addr learned my addr 172.21.15.23:0/3278078143 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740f7fe700 1 -- 172.21.15.23:0/3278078143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7400009550 con 0x7f741007be00 2023-11-16T23:50:57.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740f7fe700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7400006b20 tx=0x7f7400011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.621+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7400003b70 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.622+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7400003cd0 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.622+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741010bde0 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.622+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74000247c0 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.622+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741010bc50 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.623+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f740001a430 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.623+0000 7f740cff9700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 0x7f73f803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.623+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f74000501c0 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.624+0000 7f740effd700 1 -- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 msgr2=0x7f73f803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.623+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7410062120 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.624+0000 7f740effd700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 0x7f73f803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.629+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7400052060 con 0x7f741007be00 2023-11-16T23:50:57.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.825+0000 7f740effd700 1 -- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 msgr2=0x7f73f803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.825+0000 7f740effd700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 0x7f73f803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.895+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f741010bf70 con 0x7f741007be00 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.896+0000 7f740cff9700 1 -- 172.21.15.23:0/3278078143 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7400023550 con 0x7f741007be00 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 msgr2=0x7f73f803acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 0x7f73f803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 msgr2=0x7f741019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7400006b20 tx=0x7f7400011ad0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 shutdown_connections 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f73f8038820 0x7f73f803acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 --2- 172.21.15.23:0/3278078143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f741007be00 0x7f741019b7c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 >> 172.21.15.23:0/3278078143 conn(0x7f7410100fa0 msgr2=0x7f7410101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 shutdown_connections 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:56.898+0000 7f7415ae7700 1 -- 172.21.15.23:0/3278078143 wait complete. 2023-11-16T23:50:57.273 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2023-11-16T23:50:57.274 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2023-11-16T23:50:57.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:57.179+0000 7f2a062b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T23:50:57.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:57.320+0000 7f2a062b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T23:50:57.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:50:57 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3278078143' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T23:50:58.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:57.931+0000 7f2a062b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T23:50:59.017 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:58.698+0000 7f2a062b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T23:50:59.018 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:58.911+0000 7f2a062b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T23:51:00.248 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:50:59 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:50:59.889+0000 7f2a062b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:51:00.249 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:00.013+0000 7f2a062b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T23:51:00.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:00.243+0000 7f2a062b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:51:00.903 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:00.617+0000 7f2a062b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T23:51:01.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:00.901+0000 7f2a062b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T23:51:01.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:01.099+0000 7f2a062b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T23:51:01.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:01.455+0000 7f2a062b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T23:51:02.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:02.418+0000 7f2a062b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T23:51:02.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:02.596+0000 7f2a062b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T23:51:03.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:02.851+0000 7f2a062b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T23:51:03.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:03.004+0000 7f2a062b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T23:51:03.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:03.290+0000 7f2a062b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T23:51:04.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:04.388+0000 7f2a062b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: Active manager daemon x restarted 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: Activating manager daemon x 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: osdmap e2: 0 total, 0 up, 0 in 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: mgrmap e6: x(active, starting, since 0.0124275s) 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: Manager daemon x is now available 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:04 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:05.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: Found migration_current of "None". Setting to last migration. 2023-11-16T23:51:05.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:05.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:05.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:05.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T23:51:05.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T23:51:05.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:05.828 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:05.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:05 smithi023 ceph-mon[97824]: mgrmap e7: x(active, since 1.01105s) 2023-11-16T23:51:05.880 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.872+0000 7f9868ae8700 1 Processor -- start 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.872+0000 7f9868ae8700 1 -- start start 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.873+0000 7f9868ae8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.873+0000 7f9868ae8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864106260 con 0x7f9864105910 2023-11-16T23:51:05.889 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.873+0000 7f986259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.873+0000 7f986259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51180/0 (socket says 172.21.15.23:51180) 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.873+0000 7f986259c700 1 -- 172.21.15.23:0/3671241598 learned_addr learned my addr 172.21.15.23:0/3671241598 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.874+0000 7f986259c700 1 -- 172.21.15.23:0/3671241598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98641063a0 con 0x7f9864105910 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.874+0000 7f986259c700 1 --2- 172.21.15.23:0/3671241598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98540098a0 tx=0x7f9854009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f176fcb3b1ed5f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.874+0000 7f986159a700 1 -- 172.21.15.23:0/3671241598 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9854004160 con 0x7f9864105910 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.874+0000 7f986159a700 1 -- 172.21.15.23:0/3671241598 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98540042c0 con 0x7f9864105910 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.874+0000 7f986159a700 1 -- 172.21.15.23:0/3671241598 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9854011a00 con 0x7f9864105910 2023-11-16T23:51:05.890 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 -- 172.21.15.23:0/3671241598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 msgr2=0x7f9864105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 --2- 172.21.15.23:0/3671241598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98540098a0 tx=0x7f9854009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 -- 172.21.15.23:0/3671241598 shutdown_connections 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 --2- 172.21.15.23:0/3671241598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f9864105d20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 -- 172.21.15.23:0/3671241598 >> 172.21.15.23:0/3671241598 conn(0x7f9864100fa0 msgr2=0x7f98641033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 -- 172.21.15.23:0/3671241598 shutdown_connections 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.875+0000 7f9868ae8700 1 -- 172.21.15.23:0/3671241598 wait complete. 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.876+0000 7f9868ae8700 1 Processor -- start 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.876+0000 7f9868ae8700 1 -- start start 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.876+0000 7f9868ae8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:05.891 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.876+0000 7f9868ae8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986419bc90 con 0x7f9864105910 2023-11-16T23:51:05.892 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f986259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:05.892 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f986259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51188/0 (socket says 172.21.15.23:51188) 2023-11-16T23:51:05.892 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f986259c700 1 -- 172.21.15.23:0/3202061282 learned_addr learned my addr 172.21.15.23:0/3202061282 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f986259c700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854009550 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f986259c700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9854006af0 tx=0x7f9854011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.877+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9854003a40 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.878+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9854003ba0 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.878+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f985401a6e0 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.878+0000 7f9868ae8700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9864107960 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.878+0000 7f9868ae8700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98641077d0 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9854024a60 con 0x7f9864105910 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f98537fe700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f984c03b3e0 con 0x7f984c038820 2023-11-16T23:51:05.893 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f985404c6e0 con 0x7f9864105910 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:57.879+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:58.080+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:58.080+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:58.481+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:58.481+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:59.282+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:50:59.282+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:00.883+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:00.883+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:04.087+0000 7f9861d9b700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1477838149 2023-11-16T23:51:05.894 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:04.087+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:04.401+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f9854024610 con 0x7f9864105910 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:04.401+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 msgr2=0x7f984c03acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:04.401+0000 7f98537fe700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.398+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f985404e250 con 0x7f9864105910 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.398+0000 7f98537fe700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 0x7f984c03e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.399+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f984c03b3e0 con 0x7f984c03bfb0 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.399+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 0x7f984c03e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.400+0000 7f9861d9b700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 0x7f984c03e390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9858003a10 tx=0x7f98580092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.401+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f984c03b3e0 con 0x7f984c03bfb0 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.408+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f986419ca90 con 0x7f984c03bfb0 2023-11-16T23:51:05.895 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.408+0000 7f98537fe700 1 -- 172.21.15.23:0/3202061282 <== mgr.14118 v2:172.21.15.23:6800/3289144239 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f986419ca90 con 0x7f984c03bfb0 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 msgr2=0x7f984c03e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 0x7f984c03e390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9858003a10 tx=0x7f98580092b0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 msgr2=0x7f986419b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9854006af0 tx=0x7f9854011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 shutdown_connections 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f984c03bfb0 0x7f984c03e390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9864105910 0x7f986419b750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 --2- 172.21.15.23:0/3202061282 >> [v2:172.21.15.23:6800/1477838149,v1:172.21.15.23:6801/1477838149] conn(0x7f984c038820 0x7f984c03acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 >> 172.21.15.23:0/3202061282 conn(0x7f9864100fa0 msgr2=0x7f9864103380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 shutdown_connections 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:05.409+0000 7f98517fa700 1 -- 172.21.15.23:0/3202061282 wait complete. 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2023-11-16T23:51:05.896 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2023-11-16T23:51:06.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:06 smithi023 ceph-mon[97824]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T23:51:06.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:06 smithi023 ceph-mon[97824]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.406+0000 7f7fd1fe7700 1 Processor -- start 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.406+0000 7f7fd1fe7700 1 -- start start 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.406+0000 7f7fd1fe7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.406+0000 7f7fd1fe7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc07a820 con 0x7f7fcc07bde0 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.407+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.407+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48908/0 (socket says 172.21.15.23:48908) 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.407+0000 7f7fcb7fe700 1 -- 172.21.15.23:0/2034776035 learned_addr learned my addr 172.21.15.23:0/2034776035 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:07.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.407+0000 7f7fcb7fe700 1 -- 172.21.15.23:0/2034776035 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc07a960 con 0x7f7fcc07bde0 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fcb7fe700 1 --2- 172.21.15.23:0/2034776035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0098a0 tx=0x7f7fbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ee4c35d3e2101e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fca7fc700 1 -- 172.21.15.23:0/2034776035 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fbc004160 con 0x7f7fcc07bde0 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fca7fc700 1 -- 172.21.15.23:0/2034776035 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fbc0042c0 con 0x7f7fcc07bde0 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fca7fc700 1 -- 172.21.15.23:0/2034776035 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fbc011a00 con 0x7f7fcc07bde0 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2034776035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 msgr2=0x7f7fcc07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.408+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2034776035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0098a0 tx=0x7f7fbc009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.409+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2034776035 shutdown_connections 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.409+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2034776035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc07a2e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.409+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2034776035 >> 172.21.15.23:0/2034776035 conn(0x7f7fcc100fa0 msgr2=0x7f7fcc1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.409+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2034776035 shutdown_connections 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.409+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2034776035 wait complete. 2023-11-16T23:51:07.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.410+0000 7f7fd1fe7700 1 Processor -- start 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.410+0000 7f7fd1fe7700 1 -- start start 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.410+0000 7f7fd1fe7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.410+0000 7f7fd1fe7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19bcc0 con 0x7f7fcc07bde0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48918/0 (socket says 172.21.15.23:48918) 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fcb7fe700 1 -- 172.21.15.23:0/2693140434 learned_addr learned my addr 172.21.15.23:0/2693140434 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fcb7fe700 1 -- 172.21.15.23:0/2693140434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc009550 con 0x7f7fcc07bde0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fcb7fe700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc006af0 tx=0x7f7fbc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fbc003a40 con 0x7f7fcc07bde0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fbc003ba0 con 0x7f7fcc07bde0 2023-11-16T23:51:07.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.411+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fbc01a700 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.412+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc10be00 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.412+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc10bc70 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.413+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f7fbc024ad0 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.413+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc062120 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.413+0000 7f7fc8ff9700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 0x7f7fb403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.413+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7fbc04c500 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.414+0000 7f7fcaffd700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 0x7f7fb403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.414+0000 7f7fcaffd700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 0x7f7fb403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0006fd0 tx=0x7f7fc0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.419+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7fbc018b80 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.448+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7fbc02d910 con 0x7f7fcc07bde0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.642+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fcc102390 con 0x7f7fb40386f0 2023-11-16T23:51:07.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.652+0000 7f7fc8ff9700 1 -- 172.21.15.23:0/2693140434 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7fcc102390 con 0x7f7fb40386f0 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 msgr2=0x7f7fb403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 0x7f7fb403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0006fd0 tx=0x7f7fc0006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 msgr2=0x7f7fcc19b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc006af0 tx=0x7f7fbc011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 shutdown_connections 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7fb40386f0 0x7f7fb403aba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 --2- 172.21.15.23:0/2693140434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fcc07bde0 0x7f7fcc19b780 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 >> 172.21.15.23:0/2693140434 conn(0x7f7fcc100fa0 msgr2=0x7f7fcc101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 shutdown_connections 2023-11-16T23:51:07.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:06.655+0000 7f7fd1fe7700 1 -- 172.21.15.23:0/2693140434 wait complete. 2023-11-16T23:51:07.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: mgrmap e8: x(active, since 2s) 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:06] ENGINE Bus STARTING 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:06] ENGINE Serving on http://172.21.15.23:8765 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:07 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:08.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.688+0000 7fbde203c700 1 Processor -- start 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.688+0000 7fbde203c700 1 -- start start 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.688+0000 7fbde203c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.688+0000 7fbde203c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc07a820 con 0x7fbddc07bde0 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.688+0000 7fbde103a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbde103a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48922/0 (socket says 172.21.15.23:48922) 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbde103a700 1 -- 172.21.15.23:0/1788571994 learned_addr learned my addr 172.21.15.23:0/1788571994 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbde103a700 1 -- 172.21.15.23:0/1788571994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc07a960 con 0x7fbddc07bde0 2023-11-16T23:51:08.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbde103a700 1 --2- 172.21.15.23:0/1788571994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80098a0 tx=0x7fbdd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4221bd4e634dc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbdd3fff700 1 -- 172.21.15.23:0/1788571994 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdd8004160 con 0x7fbddc07bde0 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbdd3fff700 1 -- 172.21.15.23:0/1788571994 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdd80042c0 con 0x7fbddc07bde0 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.689+0000 7fbdd3fff700 1 -- 172.21.15.23:0/1788571994 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdd8011a00 con 0x7fbddc07bde0 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 -- 172.21.15.23:0/1788571994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 msgr2=0x7fbddc07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 --2- 172.21.15.23:0/1788571994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80098a0 tx=0x7fbdd8009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 -- 172.21.15.23:0/1788571994 shutdown_connections 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 --2- 172.21.15.23:0/1788571994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc07a2e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 -- 172.21.15.23:0/1788571994 >> 172.21.15.23:0/1788571994 conn(0x7fbddc100dc0 msgr2=0x7fbddc1031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 -- 172.21.15.23:0/1788571994 shutdown_connections 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.690+0000 7fbde203c700 1 -- 172.21.15.23:0/1788571994 wait complete. 2023-11-16T23:51:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.691+0000 7fbde203c700 1 Processor -- start 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.691+0000 7fbde203c700 1 -- start start 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde203c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde203c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc19bb90 con 0x7fbddc07bde0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde103a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde103a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48926/0 (socket says 172.21.15.23:48926) 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde103a700 1 -- 172.21.15.23:0/3186274099 learned_addr learned my addr 172.21.15.23:0/3186274099 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde103a700 1 -- 172.21.15.23:0/3186274099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009550 con 0x7fbddc07bde0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.692+0000 7fbde103a700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009fb0 tx=0x7fbdd8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.693+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdd8003a40 con 0x7fbddc07bde0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.693+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdd8003ba0 con 0x7fbddc07bde0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.693+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc10bc40 con 0x7fbddc07bde0 2023-11-16T23:51:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.693+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdd801a750 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.693+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc10bab0 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.694+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc10bdd0 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.694+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fbdd8024b60 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.694+0000 7fbdd27fc700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 0x7fbdc803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.694+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbdd804c700 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.695+0000 7fbde0839700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 0x7fbdc803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.695+0000 7fbde0839700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 0x7fbdc803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc006fd0 tx=0x7fbdcc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.700+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbdd8024430 con 0x7fbddc07bde0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.907+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fbddc102220 con 0x7fbdc80387d0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.908+0000 7fbdd27fc700 1 -- 172.21.15.23:0/3186274099 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fbddc102220 con 0x7fbdc80387d0 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 msgr2=0x7fbdc803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 0x7fbdc803ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc006fd0 tx=0x7fbdcc006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:08.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 msgr2=0x7fbddc19b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009fb0 tx=0x7fbdd8011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 shutdown_connections 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fbdc80387d0 0x7fbdc803ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 --2- 172.21.15.23:0/3186274099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbddc07bde0 0x7fbddc19b650 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 >> 172.21.15.23:0/3186274099 conn(0x7fbddc100dc0 msgr2=0x7fbddc101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 shutdown_connections 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:07.910+0000 7fbde203c700 1 -- 172.21.15.23:0/3186274099 wait complete. 2023-11-16T23:51:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2023-11-16T23:51:08.611 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:08 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:06] ENGINE Serving on https://172.21.15.23:7150 2023-11-16T23:51:08.611 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:08 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:06] ENGINE Bus STARTED 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-mon[97824]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: Generating public/private rsa key pair. 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: Your identification has been saved in /tmp/tmprj9ro3oe/key. 2023-11-16T23:51:09.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: Your public key has been saved in /tmp/tmprj9ro3oe/key.pub. 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: The key fingerprint is: 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: SHA256:O9MvcrvVNPJI2UVjxVNXkIRswy52WSoFurRiVhLQSug ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: The key's randomart image is: 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: +---[RSA 3072]----+ 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | ..o. .+ oo*X| 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | . . .. . B +o+| 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | . . .. + + = o| 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | E . + o+ =o . | 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | + S. ++ + | 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | o . o . * . | 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | + . o o | 2023-11-16T23:51:09.769 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | .ooo | 2023-11-16T23:51:09.770 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: | oo+. | 2023-11-16T23:51:09.770 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: +----[SHA256]-----+ 2023-11-16T23:51:09.848 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.836+0000 7f86e3fff700 1 Processor -- start 2023-11-16T23:51:09.848 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.836+0000 7f86e3fff700 1 -- start start 2023-11-16T23:51:09.848 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.836+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.836+0000 7f86e3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e407a820 con 0x7f86e407bde0 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48934/0 (socket says 172.21.15.23:48934) 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e2ffd700 1 -- 172.21.15.23:0/3809914029 learned_addr learned my addr 172.21.15.23:0/3809914029 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e2ffd700 1 -- 172.21.15.23:0/3809914029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e407a960 con 0x7f86e407bde0 2023-11-16T23:51:09.849 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e2ffd700 1 --2- 172.21.15.23:0/3809914029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea988ab3df3c2c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.837+0000 7f86e1ffb700 1 -- 172.21.15.23:0/3809914029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86d4004160 con 0x7f86e407bde0 2023-11-16T23:51:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e1ffb700 1 -- 172.21.15.23:0/3809914029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d40042c0 con 0x7f86e407bde0 2023-11-16T23:51:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e1ffb700 1 -- 172.21.15.23:0/3809914029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86d4011a00 con 0x7f86e407bde0 2023-11-16T23:51:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 -- 172.21.15.23:0/3809914029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 msgr2=0x7f86e407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:09.850 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 --2- 172.21.15.23:0/3809914029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86d40098a0 tx=0x7f86d4009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 -- 172.21.15.23:0/3809914029 shutdown_connections 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 --2- 172.21.15.23:0/3809914029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e407a2e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 -- 172.21.15.23:0/3809914029 >> 172.21.15.23:0/3809914029 conn(0x7f86e4100dc0 msgr2=0x7f86e41031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 -- 172.21.15.23:0/3809914029 shutdown_connections 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.838+0000 7f86e3fff700 1 -- 172.21.15.23:0/3809914029 wait complete. 2023-11-16T23:51:09.851 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.839+0000 7f86e3fff700 1 Processor -- start 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.839+0000 7f86e3fff700 1 -- start start 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e419bb90 con 0x7f86e407bde0 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48948/0 (socket says 172.21.15.23:48948) 2023-11-16T23:51:09.852 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e2ffd700 1 -- 172.21.15.23:0/350808024 learned_addr learned my addr 172.21.15.23:0/350808024 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:09.853 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.840+0000 7f86e2ffd700 1 -- 172.21.15.23:0/350808024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d4009550 con 0x7f86e407bde0 2023-11-16T23:51:09.853 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86e2ffd700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009130 tx=0x7f86d4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:09.853 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86d4003ab0 con 0x7f86e407bde0 2023-11-16T23:51:09.853 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d4003c10 con 0x7f86e407bde0 2023-11-16T23:51:09.853 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86d40247d0 con 0x7f86e407bde0 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e410bc40 con 0x7f86e407bde0 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.841+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e410bab0 con 0x7f86e407bde0 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.842+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f86d401a430 con 0x7f86e407bde0 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.842+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e4062120 con 0x7f86e407bde0 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.843+0000 7f86cbfff700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 0x7f86cc03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:09.854 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.843+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f86d404c840 con 0x7f86e407bde0 2023-11-16T23:51:09.855 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.843+0000 7f86e27fc700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 0x7f86cc03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:09.855 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.843+0000 7f86e27fc700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 0x7f86cc03acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86d8006fd0 tx=0x7f86d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:09.855 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:08.848+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f86d4018b40 con 0x7f86e407bde0 2023-11-16T23:51:09.855 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.057+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f86e4102220 con 0x7f86cc038820 2023-11-16T23:51:09.855 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.360+0000 7f86cbfff700 1 -- 172.21.15.23:0/350808024 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f86e4102220 con 0x7f86cc038820 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 msgr2=0x7f86cc03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 0x7f86cc03acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86d8006fd0 tx=0x7f86d8006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 msgr2=0x7f86e419b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009130 tx=0x7f86d4011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 shutdown_connections 2023-11-16T23:51:09.856 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f86cc038820 0x7f86cc03acd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:09.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 --2- 172.21.15.23:0/350808024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86e407bde0 0x7f86e419b650 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:09.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 >> 172.21.15.23:0/350808024 conn(0x7f86e4100dc0 msgr2=0x7f86e4101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:09.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 shutdown_connections 2023-11-16T23:51:09.857 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:09.362+0000 7f86e3fff700 1 -- 172.21.15.23:0/350808024 wait complete. 2023-11-16T23:51:10.752 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:10 smithi023 ceph-mon[97824]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:10.752 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:10 smithi023 ceph-mon[97824]: Generating ssh key... 2023-11-16T23:51:11.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:51:11.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.378+0000 7f7c091e1700 1 Processor -- start 2023-11-16T23:51:11.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c091e1700 1 -- start start 2023-11-16T23:51:11.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c091e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c091e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0407a860 con 0x7f7c0407be20 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c02d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c02d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48952/0 (socket says 172.21.15.23:48952) 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.379+0000 7f7c02d9d700 1 -- 172.21.15.23:0/1188752232 learned_addr learned my addr 172.21.15.23:0/1188752232 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.380+0000 7f7c02d9d700 1 -- 172.21.15.23:0/1188752232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0407a9a0 con 0x7f7c0407be20 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.380+0000 7f7c02d9d700 1 --2- 172.21.15.23:0/1188752232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40098a0 tx=0x7f7bf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0df1c5e8c3980f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.380+0000 7f7c01d9b700 1 -- 172.21.15.23:0/1188752232 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bf4004160 con 0x7f7c0407be20 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.380+0000 7f7c01d9b700 1 -- 172.21.15.23:0/1188752232 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf40042c0 con 0x7f7c0407be20 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.380+0000 7f7c01d9b700 1 -- 172.21.15.23:0/1188752232 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bf4011a00 con 0x7f7c0407be20 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 -- 172.21.15.23:0/1188752232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 msgr2=0x7f7c0407a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 --2- 172.21.15.23:0/1188752232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40098a0 tx=0x7f7bf4009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 -- 172.21.15.23:0/1188752232 shutdown_connections 2023-11-16T23:51:11.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 --2- 172.21.15.23:0/1188752232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0407a320 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 -- 172.21.15.23:0/1188752232 >> 172.21.15.23:0/1188752232 conn(0x7f7c04100fa0 msgr2=0x7f7c041033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 -- 172.21.15.23:0/1188752232 shutdown_connections 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.381+0000 7f7c091e1700 1 -- 172.21.15.23:0/1188752232 wait complete. 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.382+0000 7f7c091e1700 1 Processor -- start 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.382+0000 7f7c091e1700 1 -- start start 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.382+0000 7f7c091e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.382+0000 7f7c091e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419bce0 con 0x7f7c0407be20 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7c02d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7c02d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48954/0 (socket says 172.21.15.23:48954) 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7c02d9d700 1 -- 172.21.15.23:0/3317122623 learned_addr learned my addr 172.21.15.23:0/3317122623 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7c02d9d700 1 -- 172.21.15.23:0/3317122623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4009550 con 0x7f7c0407be20 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7c02d9d700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4006af0 tx=0x7f7bf4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:11.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.383+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bf4003a40 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.384+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf4003ba0 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.384+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bf401a700 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.384+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0410bdc0 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.384+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0410bc30 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.385+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7bf4024ad0 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.385+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04062120 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.385+0000 7f7bebfff700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 0x7f7bec03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.385+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7bf404c780 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.386+0000 7f7c0259c700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 0x7f7bec03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.386+0000 7f7c0259c700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 0x7f7bec03acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.390+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7bf4018b80 con 0x7f7c0407be20 2023-11-16T23:51:11.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.602+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c04102380 con 0x7f7bec038820 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.603+0000 7f7bebfff700 1 -- 172.21.15.23:0/3317122623 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7c04102380 con 0x7f7bec038820 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 msgr2=0x7f7bec03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 0x7f7bec03acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 msgr2=0x7f7c0419b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4006af0 tx=0x7f7bf4011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 shutdown_connections 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f7bec038820 0x7f7bec03acd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 --2- 172.21.15.23:0/3317122623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c0407be20 0x7f7c0419b7a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:11.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 >> 172.21.15.23:0/3317122623 conn(0x7f7c04100fa0 msgr2=0x7f7c04101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:11.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 shutdown_connections 2023-11-16T23:51:11.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:10.606+0000 7f7c091e1700 1 -- 172.21.15.23:0/3317122623 wait complete. 2023-11-16T23:51:11.128 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T23:51:11.128 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2023-11-16T23:51:11.129 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2023-11-16T23:51:11.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:11 smithi023 ceph-mon[97824]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:12.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:12 smithi023 ceph-mon[97824]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:12.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:12 smithi023 ceph-mon[97824]: Deploying cephadm binary to smithi023 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fac06593700 1 Processor -- start 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fac06593700 1 -- start start 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fac06593700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fac06593700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0007a860 con 0x7fac0007be20 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fabfffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fabfffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52684/0 (socket says 172.21.15.23:52684) 2023-11-16T23:51:15.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.652+0000 7fabfffff700 1 -- 172.21.15.23:0/4216629664 learned_addr learned my addr 172.21.15.23:0/4216629664 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.653+0000 7fabfffff700 1 -- 172.21.15.23:0/4216629664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0007a9a0 con 0x7fac0007be20 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.653+0000 7fabfffff700 1 --2- 172.21.15.23:0/4216629664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fabf00098a0 tx=0x7fabf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fcd218669903cd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.653+0000 7fabfeffd700 1 -- 172.21.15.23:0/4216629664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabf0004160 con 0x7fac0007be20 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.653+0000 7fabfeffd700 1 -- 172.21.15.23:0/4216629664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabf00042c0 con 0x7fac0007be20 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.653+0000 7fabfeffd700 1 -- 172.21.15.23:0/4216629664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabf0011a00 con 0x7fac0007be20 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 -- 172.21.15.23:0/4216629664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 msgr2=0x7fac0007a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 --2- 172.21.15.23:0/4216629664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fabf00098a0 tx=0x7fabf0009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 -- 172.21.15.23:0/4216629664 shutdown_connections 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 --2- 172.21.15.23:0/4216629664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0007a320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 -- 172.21.15.23:0/4216629664 >> 172.21.15.23:0/4216629664 conn(0x7fac00100fa0 msgr2=0x7fac001033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 -- 172.21.15.23:0/4216629664 shutdown_connections 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.654+0000 7fac06593700 1 -- 172.21.15.23:0/4216629664 wait complete. 2023-11-16T23:51:15.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.655+0000 7fac06593700 1 Processor -- start 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.655+0000 7fac06593700 1 -- start start 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.655+0000 7fac06593700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.655+0000 7fac06593700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0019bcf0 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52698/0 (socket says 172.21.15.23:52698) 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfffff700 1 -- 172.21.15.23:0/1619566766 learned_addr learned my addr 172.21.15.23:0/1619566766 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfffff700 1 -- 172.21.15.23:0/1619566766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0009550 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfffff700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009fb0 tx=0x7fabf0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.656+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabf0003930 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.657+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0010bdc0 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.657+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabf0003a90 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.657+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabf00195e0 con 0x7fac0007be20 2023-11-16T23:51:15.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.657+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0010bc30 con 0x7fac0007be20 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.658+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fabf001a430 con 0x7fac0007be20 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.658+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac00062120 con 0x7fac0007be20 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.658+0000 7fabfd7fa700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 0x7fabe803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.658+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fabf004c830 con 0x7fac0007be20 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.658+0000 7fabff7fe700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 0x7fabe803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.659+0000 7fabff7fe700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 0x7fabe803ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabf4006fd0 tx=0x7fabf4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.663+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fabf0023b80 con 0x7fac0007be20 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:11.874+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7fac00102390 con 0x7fabe8038780 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.902+0000 7fabfd7fa700 1 -- 172.21.15.23:0/1619566766 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fac00102390 con 0x7fabe8038780 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 msgr2=0x7fabe803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 0x7fabe803ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabf4006fd0 tx=0x7fabf4006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 msgr2=0x7fac0019b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:15.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009fb0 tx=0x7fabf0011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 shutdown_connections 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fabe8038780 0x7fabe803ac30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 --2- 172.21.15.23:0/1619566766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac0007be20 0x7fac0019b7b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 >> 172.21.15.23:0/1619566766 conn(0x7fac00100fa0 msgr2=0x7fac00101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.905+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 shutdown_connections 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:14.906+0000 7fac06593700 1 -- 172.21.15.23:0/1619566766 wait complete. 2023-11-16T23:51:15.318 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2023-11-16T23:51:16.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:15 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:16.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:15 smithi023 ceph-mon[97824]: Added host smithi023 2023-11-16T23:51:16.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:15 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:16.519 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-11-16T23:51:16.519 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.892+0000 7feb81db9700 1 Processor -- start 2023-11-16T23:51:16.519 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.892+0000 7feb81db9700 1 -- start start 2023-11-16T23:51:16.519 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.893+0000 7feb81db9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:16.519 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.893+0000 7feb81db9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c07a820 con 0x7feb7c07bde0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.893+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52702/0 (socket says 172.21.15.23:52702) 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7b7fe700 1 -- 172.21.15.23:0/1378440394 learned_addr learned my addr 172.21.15.23:0/1378440394 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7b7fe700 1 -- 172.21.15.23:0/1378440394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c07a960 con 0x7feb7c07bde0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7b7fe700 1 --2- 172.21.15.23:0/1378440394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098a0 tx=0x7feb6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edd6e6bfb84f998 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7a7fc700 1 -- 172.21.15.23:0/1378440394 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb6c004160 con 0x7feb7c07bde0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7a7fc700 1 -- 172.21.15.23:0/1378440394 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c0042c0 con 0x7feb7c07bde0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.894+0000 7feb7a7fc700 1 -- 172.21.15.23:0/1378440394 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb6c011a00 con 0x7feb7c07bde0 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 -- 172.21.15.23:0/1378440394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 msgr2=0x7feb7c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 --2- 172.21.15.23:0/1378440394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098a0 tx=0x7feb6c009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 -- 172.21.15.23:0/1378440394 shutdown_connections 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 --2- 172.21.15.23:0/1378440394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c07a2e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:16.520 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 -- 172.21.15.23:0/1378440394 >> 172.21.15.23:0/1378440394 conn(0x7feb7c100fa0 msgr2=0x7feb7c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 -- 172.21.15.23:0/1378440394 shutdown_connections 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.895+0000 7feb81db9700 1 -- 172.21.15.23:0/1378440394 wait complete. 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.897+0000 7feb81db9700 1 Processor -- start 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.897+0000 7feb81db9700 1 -- start start 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.897+0000 7feb81db9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.897+0000 7feb81db9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c19bcc0 con 0x7feb7c07bde0 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52704/0 (socket says 172.21.15.23:52704) 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb7b7fe700 1 -- 172.21.15.23:0/1124868906 learned_addr learned my addr 172.21.15.23:0/1124868906 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb7b7fe700 1 -- 172.21.15.23:0/1124868906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009550 con 0x7feb7c07bde0 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb7b7fe700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009fb0 tx=0x7feb6c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:16.521 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb6c003930 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c003a90 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb6c0195e0 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.898+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c10be00 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.899+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c10bc70 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.900+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7feb6c01a9d0 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.899+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c062120 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.900+0000 7feb78ff9700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 0x7feb6403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.900+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb6c04c7d0 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.900+0000 7feb7affd700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 0x7feb6403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.901+0000 7feb7affd700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 0x7feb6403ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb70006fd0 tx=0x7feb70006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:15.906+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feb6c003c00 con 0x7feb7c07bde0 2023-11-16T23:51:16.522 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.127+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7feb7c102390 con 0x7feb640387d0 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.133+0000 7feb78ff9700 1 -- 172.21.15.23:0/1124868906 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7feb7c102390 con 0x7feb640387d0 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 msgr2=0x7feb6403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 0x7feb6403ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb70006fd0 tx=0x7feb70006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 msgr2=0x7feb7c19b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009fb0 tx=0x7feb6c011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 shutdown_connections 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feb640387d0 0x7feb6403ac80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 --2- 172.21.15.23:0/1124868906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb7c07bde0 0x7feb7c19b780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 >> 172.21.15.23:0/1124868906 conn(0x7feb7c100fa0 msgr2=0x7feb7c101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 shutdown_connections 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:16.136+0000 7feb81db9700 1 -- 172.21.15.23:0/1124868906 wait complete. 2023-11-16T23:51:16.523 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2023-11-16T23:51:17.504 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:17 smithi023 ceph-mon[97824]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:17.504 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:17 smithi023 ceph-mon[97824]: Saving service mon spec with placement count:5 2023-11-16T23:51:17.504 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:17 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:17.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.272+0000 7f51a45ed700 1 Processor -- start 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a45ed700 1 -- start start 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a45ed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a45ed700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c07a840 con 0x7f519c07be00 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a2389700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a2389700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52716/0 (socket says 172.21.15.23:52716) 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.273+0000 7f51a2389700 1 -- 172.21.15.23:0/3553299363 learned_addr learned my addr 172.21.15.23:0/3553299363 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.274+0000 7f51a2389700 1 -- 172.21.15.23:0/3553299363 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c07a980 con 0x7f519c07be00 2023-11-16T23:51:17.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.274+0000 7f51a2389700 1 --2- 172.21.15.23:0/3553299363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51980098a0 tx=0x7f5198009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174e33974101c9da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.274+0000 7f51a1387700 1 -- 172.21.15.23:0/3553299363 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5198004160 con 0x7f519c07be00 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.274+0000 7f51a1387700 1 -- 172.21.15.23:0/3553299363 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51980042c0 con 0x7f519c07be00 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.274+0000 7f51a1387700 1 -- 172.21.15.23:0/3553299363 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5198011a00 con 0x7f519c07be00 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 -- 172.21.15.23:0/3553299363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 msgr2=0x7f519c07a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 --2- 172.21.15.23:0/3553299363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51980098a0 tx=0x7f5198009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 -- 172.21.15.23:0/3553299363 shutdown_connections 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 --2- 172.21.15.23:0/3553299363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c07a300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 -- 172.21.15.23:0/3553299363 >> 172.21.15.23:0/3553299363 conn(0x7f519c100fa0 msgr2=0x7f519c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 -- 172.21.15.23:0/3553299363 shutdown_connections 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.275+0000 7f51a45ed700 1 -- 172.21.15.23:0/3553299363 wait complete. 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.277+0000 7f51a45ed700 1 Processor -- start 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.277+0000 7f51a45ed700 1 -- start start 2023-11-16T23:51:17.999 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.277+0000 7f51a45ed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.277+0000 7f51a45ed700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c19bce0 con 0x7f519c07be00 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.277+0000 7f51a2389700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51a2389700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52726/0 (socket says 172.21.15.23:52726) 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51a2389700 1 -- 172.21.15.23:0/4225726978 learned_addr learned my addr 172.21.15.23:0/4225726978 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51a2389700 1 -- 172.21.15.23:0/4225726978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5198009550 con 0x7f519c07be00 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51a2389700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5198006b20 tx=0x7f5198011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5198003b70 con 0x7f519c07be00 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5198003cd0 con 0x7f519c07be00 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51980247c0 con 0x7f519c07be00 2023-11-16T23:51:18.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.278+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c10bde0 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.279+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c10bc50 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.279+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f519801a430 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.280+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c062120 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.280+0000 7f51937fe700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 0x7f518803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.280+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f519804c980 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.280+0000 7f51a1b88700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 0x7f518803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.281+0000 7f51a1b88700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 0x7f518803acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f518c006fd0 tx=0x7f518c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.285+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f519801a6e0 con 0x7f519c07be00 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.504+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f519c102380 con 0x7f5188038820 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.508+0000 7f51937fe700 1 -- 172.21.15.23:0/4225726978 <== mgr.14118 v2:172.21.15.23:6800/3289144239 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f519c102380 con 0x7f5188038820 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 msgr2=0x7f518803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 0x7f518803acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f518c006fd0 tx=0x7f518c006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:18.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 msgr2=0x7f519c19b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5198006b20 tx=0x7f5198011ad0 comp rx=0 tx=0).stop 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 shutdown_connections 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f5188038820 0x7f518803acd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 --2- 172.21.15.23:0/4225726978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f519c07be00 0x7f519c19b7a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 >> 172.21.15.23:0/4225726978 conn(0x7f519c100fa0 msgr2=0x7f519c101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 shutdown_connections 2023-11-16T23:51:18.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:17.511+0000 7f51a45ed700 1 -- 172.21.15.23:0/4225726978 wait complete. 2023-11-16T23:51:18.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:18 smithi023 ceph-mon[97824]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:18.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:18 smithi023 ceph-mon[97824]: Saving service mgr spec with placement count:2 2023-11-16T23:51:18.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:18 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:18.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:18 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:18.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:18 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.499+0000 7f5105da2700 1 Processor -- start 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.499+0000 7f5105da2700 1 -- start start 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.499+0000 7f5105da2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.499+0000 7f5105da2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510007a800 con 0x7f510007bdc0 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52728/0 (socket says 172.21.15.23:52728) 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50ff7fe700 1 -- 172.21.15.23:0/903730438 learned_addr learned my addr 172.21.15.23:0/903730438 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50ff7fe700 1 -- 172.21.15.23:0/903730438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510007a940 con 0x7f510007bdc0 2023-11-16T23:51:19.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50ff7fe700 1 --2- 172.21.15.23:0/903730438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50f00098a0 tx=0x7f50f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b688b35aba23eb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.500+0000 7f50fe7fc700 1 -- 172.21.15.23:0/903730438 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f0004160 con 0x7f510007bdc0 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f50fe7fc700 1 -- 172.21.15.23:0/903730438 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f00042c0 con 0x7f510007bdc0 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f50fe7fc700 1 -- 172.21.15.23:0/903730438 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f00119b0 con 0x7f510007bdc0 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 -- 172.21.15.23:0/903730438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 msgr2=0x7f510007a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 --2- 172.21.15.23:0/903730438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50f00098a0 tx=0x7f50f0009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 -- 172.21.15.23:0/903730438 shutdown_connections 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 --2- 172.21.15.23:0/903730438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510007a2c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 -- 172.21.15.23:0/903730438 >> 172.21.15.23:0/903730438 conn(0x7f5100100fa0 msgr2=0x7f51001033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.501+0000 7f5105da2700 1 -- 172.21.15.23:0/903730438 shutdown_connections 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.502+0000 7f5105da2700 1 -- 172.21.15.23:0/903730438 wait complete. 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.502+0000 7f5105da2700 1 Processor -- start 2023-11-16T23:51:19.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f5105da2700 1 -- start start 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f5105da2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f5105da2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510019bd00 con 0x7f510007bdc0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f50ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f50ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52734/0 (socket says 172.21.15.23:52734) 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.503+0000 7f50ff7fe700 1 -- 172.21.15.23:0/3110348039 learned_addr learned my addr 172.21.15.23:0/3110348039 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f50ff7fe700 1 -- 172.21.15.23:0/3110348039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0009550 con 0x7f510007bdc0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f50ff7fe700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50f0000c00 tx=0x7f50f0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f00039f0 con 0x7f510007bdc0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f0003b50 con 0x7f510007bdc0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50f001a780 con 0x7f510007bdc0 2023-11-16T23:51:19.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510010be10 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.504+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510010bc80 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.505+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f50f0024440 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.505+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510010bfa0 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.506+0000 7f50fcff9700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 0x7f50e803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.506+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f50f004c7b0 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.506+0000 7f50feffd700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 0x7f50e803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.507+0000 7f50feffd700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 0x7f50e803acd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50f4006fd0 tx=0x7f50f4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.511+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f50f00246f0 con 0x7f510007bdc0 2023-11-16T23:51:19.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.718+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5100062120 con 0x7f510007bdc0 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.722+0000 7f50fcff9700 1 -- 172.21.15.23:0/3110348039 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f50f0018350 con 0x7f510007bdc0 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.724+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 msgr2=0x7f50e803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.724+0000 7f5105da2700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 0x7f50e803acd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50f4006fd0 tx=0x7f50f4006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.724+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 msgr2=0x7f510019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.724+0000 7f5105da2700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50f0000c00 tx=0x7f50f0011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 shutdown_connections 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f50e8038820 0x7f50e803acd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 --2- 172.21.15.23:0/3110348039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f510007bdc0 0x7f510019b7c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 >> 172.21.15.23:0/3110348039 conn(0x7f5100100fa0 msgr2=0x7f5100101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 shutdown_connections 2023-11-16T23:51:19.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:18.725+0000 7f5105da2700 1 -- 172.21.15.23:0/3110348039 wait complete. 2023-11-16T23:51:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3110348039' entity='client.admin' 2023-11-16T23:51:20.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f72168f7700 1 Processor -- start 2023-11-16T23:51:20.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f72168f7700 1 -- start start 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f72168f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f72168f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721007a800 con 0x7f721007bdc0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f720ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.676+0000 7f720ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52738/0 (socket says 172.21.15.23:52738) 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720ffff700 1 -- 172.21.15.23:0/1368813570 learned_addr learned my addr 172.21.15.23:0/1368813570 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720ffff700 1 -- 172.21.15.23:0/1368813570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721007a940 con 0x7f721007bdc0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720ffff700 1 --2- 172.21.15.23:0/1368813570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72000098a0 tx=0x7f7200009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd7df3e2ba3e5e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720effd700 1 -- 172.21.15.23:0/1368813570 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7200004160 con 0x7f721007bdc0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720effd700 1 -- 172.21.15.23:0/1368813570 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72000042c0 con 0x7f721007bdc0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.677+0000 7f720effd700 1 -- 172.21.15.23:0/1368813570 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72000119b0 con 0x7f721007bdc0 2023-11-16T23:51:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 -- 172.21.15.23:0/1368813570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 msgr2=0x7f721007a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 --2- 172.21.15.23:0/1368813570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72000098a0 tx=0x7f7200009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 -- 172.21.15.23:0/1368813570 shutdown_connections 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 --2- 172.21.15.23:0/1368813570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721007a2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 -- 172.21.15.23:0/1368813570 >> 172.21.15.23:0/1368813570 conn(0x7f7210100fa0 msgr2=0x7f72101033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 -- 172.21.15.23:0/1368813570 shutdown_connections 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.678+0000 7f72168f7700 1 -- 172.21.15.23:0/1368813570 wait complete. 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.679+0000 7f72168f7700 1 Processor -- start 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.679+0000 7f72168f7700 1 -- start start 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f72168f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f72168f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721019bcd0 con 0x7f721007bdc0 2023-11-16T23:51:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f720ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f720ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52750/0 (socket says 172.21.15.23:52750) 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f720ffff700 1 -- 172.21.15.23:0/3203655307 learned_addr learned my addr 172.21.15.23:0/3203655307 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.680+0000 7f720ffff700 1 -- 172.21.15.23:0/3203655307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7200009550 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f720ffff700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c00 tx=0x7f7200011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72000039f0 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7200003b50 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f720001a780 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721010be10 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.681+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721010bc80 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.682+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7200024440 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.682+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721010bfa0 con 0x7f721007bdc0 2023-11-16T23:51:20.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.682+0000 7f720d7fa700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 0x7f71f803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.682+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f720004c830 con 0x7f721007bdc0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.683+0000 7f720f7fe700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 0x7f71f803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.683+0000 7f720f7fe700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 0x7f71f803acd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7204006fd0 tx=0x7f7204006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.687+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f72000246f0 con 0x7f721007bdc0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.902+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f721019f410 con 0x7f721007bdc0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.905+0000 7f720d7fa700 1 -- 172.21.15.23:0/3203655307 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7200018350 con 0x7f721007bdc0 2023-11-16T23:51:20.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.908+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 msgr2=0x7f71f803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.908+0000 7f72168f7700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 0x7f71f803acd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7204006fd0 tx=0x7f7204006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.908+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 msgr2=0x7f721019b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.908+0000 7f72168f7700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c00 tx=0x7f7200011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 shutdown_connections 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f71f8038820 0x7f71f803acd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 --2- 172.21.15.23:0/3203655307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f721007bdc0 0x7f721019b790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 >> 172.21.15.23:0/3203655307 conn(0x7f7210100fa0 msgr2=0x7f7210101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 shutdown_connections 2023-11-16T23:51:20.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:19.909+0000 7f72168f7700 1 -- 172.21.15.23:0/3203655307 wait complete. 2023-11-16T23:51:20.312 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2023-11-16T23:51:21.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:20 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3203655307' entity='client.admin' 2023-11-16T23:51:22.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-mon[97824]: from='mgr.14118 172.21.15.23:0/2826706699' entity='mgr.x' 2023-11-16T23:51:22.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3704495314' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T23:51:22.428 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: ignoring --setuser ceph since I am not root 2023-11-16T23:51:22.428 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: ignoring --setgroup ceph since I am not root 2023-11-16T23:51:22.428 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:22.427+0000 7f3b572a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T23:51:22.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.014+0000 7f51d7e8b700 1 Processor -- start 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.015+0000 7f51d7e8b700 1 -- start start 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.015+0000 7f51d7e8b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.015+0000 7f51d7e8b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d007a840 con 0x7f51d007be00 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.015+0000 7f51d5c27700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d5c27700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40386/0 (socket says 172.21.15.23:40386) 2023-11-16T23:51:22.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d5c27700 1 -- 172.21.15.23:0/1213779471 learned_addr learned my addr 172.21.15.23:0/1213779471 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d5c27700 1 -- 172.21.15.23:0/1213779471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d007a980 con 0x7f51d007be00 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d5c27700 1 --2- 172.21.15.23:0/1213779471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0098a0 tx=0x7f51cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2518fe02835ea70 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d4c25700 1 -- 172.21.15.23:0/1213779471 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51cc004160 con 0x7f51d007be00 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.016+0000 7f51d4c25700 1 -- 172.21.15.23:0/1213779471 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51cc0042c0 con 0x7f51d007be00 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d4c25700 1 -- 172.21.15.23:0/1213779471 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51cc0119b0 con 0x7f51d007be00 2023-11-16T23:51:22.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 -- 172.21.15.23:0/1213779471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 msgr2=0x7f51d007a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/1213779471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0098a0 tx=0x7f51cc009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 -- 172.21.15.23:0/1213779471 shutdown_connections 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/1213779471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d007a300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 -- 172.21.15.23:0/1213779471 >> 172.21.15.23:0/1213779471 conn(0x7f51d0100fa0 msgr2=0x7f51d01033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 -- 172.21.15.23:0/1213779471 shutdown_connections 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.017+0000 7f51d7e8b700 1 -- 172.21.15.23:0/1213779471 wait complete. 2023-11-16T23:51:22.595 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.018+0000 7f51d7e8b700 1 Processor -- start 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d7e8b700 1 -- start start 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d7e8b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d7e8b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d019bcf0 con 0x7f51d007be00 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d5c27700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d5c27700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40392/0 (socket says 172.21.15.23:40392) 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.019+0000 7f51d5c27700 1 -- 172.21.15.23:0/3704495314 learned_addr learned my addr 172.21.15.23:0/3704495314 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51d5c27700 1 -- 172.21.15.23:0/3704495314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc009550 con 0x7f51d007be00 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51d5c27700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51cc003e60 tx=0x7f51cc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51cc003b20 con 0x7f51d007be00 2023-11-16T23:51:22.596 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51cc003c80 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51cc024760 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d010be30 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.020+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d010bca0 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.021+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f51cc01a430 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.021+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d010bfc0 con 0x7f51d007be00 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.022+0000 7f51c6ffd700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 0x7f51bc03acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:22.597 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.022+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f51cc04ca30 con 0x7f51d007be00 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.022+0000 7f51d5426700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 0x7f51bc03acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.022+0000 7f51d5426700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 0x7f51bc03acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006fd0 tx=0x7f51c0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.027+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f51cc018b40 con 0x7f51d007be00 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:21.309+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f51d019f3c0 con 0x7f51d007be00 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.127+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f51cc01abc0 con 0x7f51d007be00 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.129+0000 7f51c6ffd700 1 -- 172.21.15.23:0/3704495314 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f51cc01a6e0 con 0x7f51d007be00 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.131+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 msgr2=0x7f51bc03acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:22.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.131+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 0x7f51bc03acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f51c0006fd0 tx=0x7f51c0006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.131+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 msgr2=0x7f51d019b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.131+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51cc003e60 tx=0x7f51cc011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 shutdown_connections 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7f51bc038820 0x7f51bc03acd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 --2- 172.21.15.23:0/3704495314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51d007be00 0x7f51d019b7b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 >> 172.21.15.23:0/3704495314 conn(0x7f51d0100fa0 msgr2=0x7f51d0101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 shutdown_connections 2023-11-16T23:51:22.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:22.132+0000 7f51d7e8b700 1 -- 172.21.15.23:0/3704495314 wait complete. 2023-11-16T23:51:22.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:22.537+0000 7f3b572a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T23:51:22.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:22.660+0000 7f3b572a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T23:51:23.130 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:22.787+0000 7f3b572a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T23:51:23.130 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:23.026+0000 7f3b572a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T23:51:23.436 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3704495314' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T23:51:23.436 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-mon[97824]: mgrmap e9: x(active, since 17s) 2023-11-16T23:51:23.739 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:23.435+0000 7f3b572a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T23:51:23.739 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:23.566+0000 7f3b572a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T23:51:23.739 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:23.738+0000 7f3b572a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T23:51:23.882 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:51:23.882 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-11-16T23:51:23.882 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T23:51:23.882 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.172+0000 7fef1882c700 1 Processor -- start 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.172+0000 7fef1882c700 1 -- start start 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.172+0000 7fef1882c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.172+0000 7fef1882c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1407a880 con 0x7fef1407be40 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.173+0000 7fef1259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.173+0000 7fef1259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40424/0 (socket says 172.21.15.23:40424) 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.173+0000 7fef1259c700 1 -- 172.21.15.23:0/1440236828 learned_addr learned my addr 172.21.15.23:0/1440236828 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.173+0000 7fef1259c700 1 -- 172.21.15.23:0/1440236828 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1407a9c0 con 0x7fef1407be40 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.173+0000 7fef1259c700 1 --2- 172.21.15.23:0/1440236828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef040098a0 tx=0x7fef04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5f3c2a9b52249b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1159a700 1 -- 172.21.15.23:0/1440236828 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef04004160 con 0x7fef1407be40 2023-11-16T23:51:23.883 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1159a700 1 -- 172.21.15.23:0/1440236828 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef040042c0 con 0x7fef1407be40 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1159a700 1 -- 172.21.15.23:0/1440236828 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef040119b0 con 0x7fef1407be40 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1882c700 1 -- 172.21.15.23:0/1440236828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 msgr2=0x7fef1407a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1882c700 1 --2- 172.21.15.23:0/1440236828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef040098a0 tx=0x7fef04009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.174+0000 7fef1882c700 1 -- 172.21.15.23:0/1440236828 shutdown_connections 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.175+0000 7fef1882c700 1 --2- 172.21.15.23:0/1440236828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1407a340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.175+0000 7fef1882c700 1 -- 172.21.15.23:0/1440236828 >> 172.21.15.23:0/1440236828 conn(0x7fef14100fa0 msgr2=0x7fef141033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.175+0000 7fef1882c700 1 -- 172.21.15.23:0/1440236828 shutdown_connections 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.175+0000 7fef1882c700 1 -- 172.21.15.23:0/1440236828 wait complete. 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1882c700 1 Processor -- start 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1882c700 1 -- start start 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1882c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1882c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1419bcf0 con 0x7fef1407be40 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:23.884 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40436/0 (socket says 172.21.15.23:40436) 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.176+0000 7fef1259c700 1 -- 172.21.15.23:0/2413799161 learned_addr learned my addr 172.21.15.23:0/2413799161 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef1259c700 1 -- 172.21.15.23:0/2413799161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef04009550 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef1259c700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef04000c00 tx=0x7fef04011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef040039f0 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef04003b50 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef0401a780 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1410bdf0 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.177+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1410bc60 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.179+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1410bf80 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.179+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fef04024440 con 0x7fef1407be40 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.179+0000 7fef037fe700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 0x7feefc03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:23.885 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.179+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fef0404c7e0 con 0x7fef1407be40 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.179+0000 7fef11d9b700 1 -- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 msgr2=0x7feefc03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.180+0000 7fef11d9b700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 0x7feefc03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.184+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fef04018db0 con 0x7fef1407be40 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.381+0000 7fef11d9b700 1 -- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 msgr2=0x7feefc03ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.381+0000 7fef11d9b700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 0x7feefc03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.447+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fef1419f3e0 con 0x7fef1407be40 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.447+0000 7fef037fe700 1 -- 172.21.15.23:0/2413799161 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fef0404c1b0 con 0x7fef1407be40 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 msgr2=0x7feefc03ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 0x7feefc03ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 msgr2=0x7fef1419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef04000c00 tx=0x7fef04011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 shutdown_connections 2023-11-16T23:51:23.886 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7feefc038870 0x7feefc03ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 --2- 172.21.15.23:0/2413799161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1407be40 0x7fef1419b7b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 >> 172.21.15.23:0/2413799161 conn(0x7fef14100fa0 msgr2=0x7fef14101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 shutdown_connections 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:23.450+0000 7fef1882c700 1 -- 172.21.15.23:0/2413799161 wait complete. 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2023-11-16T23:51:23.887 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2023-11-16T23:51:24.017 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:23 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:23.871+0000 7f3b572a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T23:51:24.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:24 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2413799161' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T23:51:24.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:24 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:24.461+0000 7f3b572a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T23:51:25.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:25 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:25.235+0000 7f3b572a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T23:51:25.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:25 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:25.450+0000 7f3b572a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T23:51:26.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:26 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:26.433+0000 7f3b572a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:51:26.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:26 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:26.557+0000 7f3b572a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T23:51:27.163 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:26 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:26.787+0000 7f3b572a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:51:27.447 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:27 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:27.162+0000 7f3b572a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T23:51:27.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:27 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:27.447+0000 7f3b572a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T23:51:27.832 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:27 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:27.647+0000 7f3b572a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T23:51:28.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:28 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:28.003+0000 7f3b572a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T23:51:29.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:28 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:28.972+0000 7f3b572a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T23:51:29.268 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:29 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:29.153+0000 7f3b572a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T23:51:29.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:29 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:29.411+0000 7f3b572a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T23:51:29.767 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:29 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:29.565+0000 7f3b572a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T23:51:30.267 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:29 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:29.854+0000 7f3b572a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T23:51:31.268 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:51:30 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:51:30.955+0000 7f3b572a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T23:51:31.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: Active manager daemon x restarted 2023-11-16T23:51:31.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: Activating manager daemon x 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: osdmap e3: 0 total, 0 up, 0 in 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: mgrmap e10: x(active, starting, since 0.00856513s) 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T23:51:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T23:51:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T23:51:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: Manager daemon x is now available 2023-11-16T23:51:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:32.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T23:51:32.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T23:51:32.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:32 smithi023 ceph-mon[97824]: mgrmap e11: x(active, since 1.01079s) 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.395+0000 7fa635f41700 1 Processor -- start 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa635f41700 1 -- start start 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa635f41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:32.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa635f41700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63007a800 con 0x7fa63007bdc0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40450/0 (socket says 172.21.15.23:40450) 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.396+0000 7fa62f7fe700 1 -- 172.21.15.23:0/2868321064 learned_addr learned my addr 172.21.15.23:0/2868321064 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.397+0000 7fa62f7fe700 1 -- 172.21.15.23:0/2868321064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63007a940 con 0x7fa63007bdc0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.397+0000 7fa62f7fe700 1 --2- 172.21.15.23:0/2868321064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6200098a0 tx=0x7fa620009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53a824a9399343e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.397+0000 7fa62e7fc700 1 -- 172.21.15.23:0/2868321064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa620004160 con 0x7fa63007bdc0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.397+0000 7fa62e7fc700 1 -- 172.21.15.23:0/2868321064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6200042c0 con 0x7fa63007bdc0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.397+0000 7fa62e7fc700 1 -- 172.21.15.23:0/2868321064 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6200119b0 con 0x7fa63007bdc0 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 -- 172.21.15.23:0/2868321064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 msgr2=0x7fa63007a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 --2- 172.21.15.23:0/2868321064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6200098a0 tx=0x7fa620009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 -- 172.21.15.23:0/2868321064 shutdown_connections 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 --2- 172.21.15.23:0/2868321064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63007a2c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 -- 172.21.15.23:0/2868321064 >> 172.21.15.23:0/2868321064 conn(0x7fa630100fa0 msgr2=0x7fa6301033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 -- 172.21.15.23:0/2868321064 shutdown_connections 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.398+0000 7fa635f41700 1 -- 172.21.15.23:0/2868321064 wait complete. 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.399+0000 7fa635f41700 1 Processor -- start 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.399+0000 7fa635f41700 1 -- start start 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.399+0000 7fa635f41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.399+0000 7fa635f41700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63019bd00 con 0x7fa63007bdc0 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.400+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.400+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40458/0 (socket says 172.21.15.23:40458) 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.400+0000 7fa62f7fe700 1 -- 172.21.15.23:0/2504969461 learned_addr learned my addr 172.21.15.23:0/2504969461 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:32.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.400+0000 7fa62f7fe700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620009550 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.400+0000 7fa62f7fe700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa620000c00 tx=0x7fa620011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.401+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6200039f0 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.401+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa620003b50 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.401+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa62001a780 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.401+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63010be10 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.401+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63010bc80 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.402+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fa620024440 con 0x7fa63007bdc0 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.402+0000 7fa62cff9700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.402+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa61803b430 con 0x7fa618038870 2023-11-16T23:51:32.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.402+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa62004c850 con 0x7fa63007bdc0 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.402+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.403+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.604+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:24.604+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:25.005+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:25.005+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T23:51:32.394 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:25.806+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:25.806+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:27.407+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:27.408+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:30.611+0000 7fa62effd700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3289144239 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:30.611+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:30.964+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7fa62002d910 con 0x7fa63007bdc0 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:30.964+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 msgr2=0x7fa61803ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:30.964+0000 7fa62cff9700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.965+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fa62004dde0 con 0x7fa63007bdc0 2023-11-16T23:51:32.396 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.965+0000 7fa62cff9700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 0x7fa61803e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.965+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa61803b430 con 0x7fa61803c1a0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.966+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 0x7fa61803e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.967+0000 7fa62effd700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 0x7fa61803e580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa624003a10 tx=0x7fa624009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.968+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fa61803b430 con 0x7fa61803c1a0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.980+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa63019cb00 con 0x7fa61803c1a0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.980+0000 7fa62cff9700 1 -- 172.21.15.23:0/2504969461 <== mgr.14150 v2:172.21.15.23:6800/499313593 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa63019cb00 con 0x7fa61803c1a0 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 msgr2=0x7fa61803e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 0x7fa61803e580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa624003a10 tx=0x7fa624009270 comp rx=0 tx=0).stop 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 msgr2=0x7fa63019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa620000c00 tx=0x7fa620011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 shutdown_connections 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa61803c1a0 0x7fa61803e580 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.397 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa63007bdc0 0x7fa63019b7c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 --2- 172.21.15.23:0/2504969461 >> [v2:172.21.15.23:6800/3289144239,v1:172.21.15.23:6801/3289144239] conn(0x7fa618038870 0x7fa61803ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 >> 172.21.15.23:0/2504969461 conn(0x7fa630100fa0 msgr2=0x7fa630101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 shutdown_connections 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:31.981+0000 7fa635f41700 1 -- 172.21.15.23:0/2504969461 wait complete. 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2023-11-16T23:51:32.398 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:31] ENGINE Bus STARTING 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:31] ENGINE Serving on http://172.21.15.23:8765 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:31] ENGINE Serving on https://172.21.15.23:7150 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: [16/Nov/2023:23:51:31] ENGINE Bus STARTED 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T23:51:33.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:33 smithi023 ceph-mon[97824]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T23:51:33.872 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.947+0000 7fdfb4435700 1 Processor -- start 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.947+0000 7fdfb4435700 1 -- start start 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.948+0000 7fdfb4435700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.948+0000 7fdfb4435700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac07a820 con 0x7fdfac07bde0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.948+0000 7fdfb21d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.948+0000 7fdfb21d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59958/0 (socket says 172.21.15.23:59958) 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.948+0000 7fdfb21d1700 1 -- 172.21.15.23:0/2049190102 learned_addr learned my addr 172.21.15.23:0/2049190102 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.949+0000 7fdfb21d1700 1 -- 172.21.15.23:0/2049190102 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac07a960 con 0x7fdfac07bde0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.949+0000 7fdfb21d1700 1 --2- 172.21.15.23:0/2049190102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80098a0 tx=0x7fdfa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f55fc7c0f36913 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.949+0000 7fdfb11cf700 1 -- 172.21.15.23:0/2049190102 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfa8004160 con 0x7fdfac07bde0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.949+0000 7fdfb11cf700 1 -- 172.21.15.23:0/2049190102 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfa80042c0 con 0x7fdfac07bde0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.949+0000 7fdfb11cf700 1 -- 172.21.15.23:0/2049190102 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfa80119b0 con 0x7fdfac07bde0 2023-11-16T23:51:33.873 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 -- 172.21.15.23:0/2049190102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 msgr2=0x7fdfac07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 --2- 172.21.15.23:0/2049190102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80098a0 tx=0x7fdfa8009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 -- 172.21.15.23:0/2049190102 shutdown_connections 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 --2- 172.21.15.23:0/2049190102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac07a2e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 -- 172.21.15.23:0/2049190102 >> 172.21.15.23:0/2049190102 conn(0x7fdfac100fa0 msgr2=0x7fdfac1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 -- 172.21.15.23:0/2049190102 shutdown_connections 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.950+0000 7fdfb4435700 1 -- 172.21.15.23:0/2049190102 wait complete. 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.951+0000 7fdfb4435700 1 Processor -- start 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb4435700 1 -- start start 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb4435700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb4435700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac19bc80 con 0x7fdfac07bde0 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb21d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:33.874 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb21d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59970/0 (socket says 172.21.15.23:59970) 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.952+0000 7fdfb21d1700 1 -- 172.21.15.23:0/1989650421 learned_addr learned my addr 172.21.15.23:0/1989650421 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfb21d1700 1 -- 172.21.15.23:0/1989650421 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa8009550 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfb21d1700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfa80039f0 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfa8003b50 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac10be50 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfa801a780 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.953+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac10bcc0 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.954+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fdfa801aa40 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.955+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac10bfe0 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.955+0000 7fdfa37fe700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 0x7fdf9803aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.955+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdfa804c720 con 0x7fdfac07bde0 2023-11-16T23:51:33.875 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.955+0000 7fdfb19d0700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 0x7fdf9803aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.956+0000 7fdfb19d0700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 0x7fdf9803aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:32.964+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdfa80237b0 con 0x7fdfac07bde0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.020+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fdfa802dc90 con 0x7fdfac07bde0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.209+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfac102380 con 0x7fdf980386f0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.415+0000 7fdfa37fe700 1 -- 172.21.15.23:0/1989650421 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fdfac102380 con 0x7fdf980386f0 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.417+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 msgr2=0x7fdf9803aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.417+0000 7fdfb4435700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 0x7fdf9803aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006fd0 tx=0x7fdf9c006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 msgr2=0x7fdfac19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 shutdown_connections 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf980386f0 0x7fdf9803aba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 --2- 172.21.15.23:0/1989650421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfac07bde0 0x7fdfac19b740 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:33.876 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 >> 172.21.15.23:0/1989650421 conn(0x7fdfac100fa0 msgr2=0x7fdfac101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:33.877 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 shutdown_connections 2023-11-16T23:51:33.877 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:33.418+0000 7fdfb4435700 1 -- 172.21.15.23:0/1989650421 wait complete. 2023-11-16T23:51:33.877 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2023-11-16T23:51:34.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:34 smithi023 ceph-mon[97824]: mgrmap e12: x(active, since 2s) 2023-11-16T23:51:34.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:34 smithi023 ceph-mon[97824]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:34.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:34.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:34.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:35.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.OcJvbwXepsshUyENNVB7O4j/0NyaqKUTL13c1EeuWw.naXNhOFau", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700178694, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f9613609700 1 Processor -- start 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f9613609700 1 -- start start 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f9613609700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f9613609700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c1062a0 con 0x7f960c079ce0 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f96113a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f96113a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59980/0 (socket says 172.21.15.23:59980) 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.417+0000 7f96113a5700 1 -- 172.21.15.23:0/2536406316 learned_addr learned my addr 172.21.15.23:0/2536406316 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.418+0000 7f96113a5700 1 -- 172.21.15.23:0/2536406316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c10adf0 con 0x7f960c079ce0 2023-11-16T23:51:35.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.418+0000 7f96113a5700 1 --2- 172.21.15.23:0/2536406316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96080098a0 tx=0x7f9608009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a0d30b9a7d3d20 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.418+0000 7f9603fff700 1 -- 172.21.15.23:0/2536406316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9608004160 con 0x7f960c079ce0 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.418+0000 7f9603fff700 1 -- 172.21.15.23:0/2536406316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96080042c0 con 0x7f960c079ce0 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.418+0000 7f9603fff700 1 -- 172.21.15.23:0/2536406316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96080119b0 con 0x7f960c079ce0 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 -- 172.21.15.23:0/2536406316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 msgr2=0x7f960c10a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 --2- 172.21.15.23:0/2536406316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96080098a0 tx=0x7f9608009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 -- 172.21.15.23:0/2536406316 shutdown_connections 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 --2- 172.21.15.23:0/2536406316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c10a8b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 -- 172.21.15.23:0/2536406316 >> 172.21.15.23:0/2536406316 conn(0x7f960c0ffe40 msgr2=0x7f960c102270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 -- 172.21.15.23:0/2536406316 shutdown_connections 2023-11-16T23:51:35.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.419+0000 7f9613609700 1 -- 172.21.15.23:0/2536406316 wait complete. 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.421+0000 7f9613609700 1 Processor -- start 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.421+0000 7f9613609700 1 -- start start 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.421+0000 7f9613609700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.421+0000 7f9613609700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c19bd10 con 0x7f960c079ce0 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96113a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96113a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59996/0 (socket says 172.21.15.23:59996) 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96113a5700 1 -- 172.21.15.23:0/707509977 learned_addr learned my addr 172.21.15.23:0/707509977 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96113a5700 1 -- 172.21.15.23:0/707509977 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9608009550 con 0x7f960c079ce0 2023-11-16T23:51:35.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96113a5700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9608003b20 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9608003c80 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c10bdc0 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.422+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9608019770 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.423+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c10bc30 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.424+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f960801a430 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.424+0000 7f96027fc700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 0x7f95f803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.424+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9608051660 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.424+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c10bf50 con 0x7f960c079ce0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.424+0000 7f9610ba4700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 0x7f95f803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:35.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.425+0000 7f9610ba4700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 0x7f95f803acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95fc006fd0 tx=0x7f95fc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.430+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9608052060 con 0x7f960c079ce0 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.693+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f960c19c360 con 0x7f95f8038820 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.940+0000 7f96027fc700 1 -- 172.21.15.23:0/707509977 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f960c19c360 con 0x7f95f8038820 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 msgr2=0x7f95f803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 0x7f95f803acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95fc006fd0 tx=0x7f95fc006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 msgr2=0x7f960c19b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 shutdown_connections 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f95f8038820 0x7f95f803acd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 --2- 172.21.15.23:0/707509977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f960c079ce0 0x7f960c19b7d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:35.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 >> 172.21.15.23:0/707509977 conn(0x7f960c0ffe40 msgr2=0x7f960c100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:35.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 shutdown_connections 2023-11-16T23:51:35.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:34.943+0000 7f9613609700 1 -- 172.21.15.23:0/707509977 wait complete. 2023-11-16T23:51:35.573 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2023-11-16T23:51:36.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:35 smithi023 ceph-mon[97824]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:36.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:36.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:36.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2023-11-16T23:51:36.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.138+0000 7f36446ab700 1 Processor -- start 2023-11-16T23:51:36.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f36446ab700 1 -- start start 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f36446ab700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f36446ab700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c07a880 con 0x7f363c07be40 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f3642447700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f3642447700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60012/0 (socket says 172.21.15.23:60012) 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.139+0000 7f3642447700 1 -- 172.21.15.23:0/1363949956 learned_addr learned my addr 172.21.15.23:0/1363949956 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.140+0000 7f3642447700 1 -- 172.21.15.23:0/1363949956 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c07a9c0 con 0x7f363c07be40 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.140+0000 7f3642447700 1 --2- 172.21.15.23:0/1363949956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36380098a0 tx=0x7f3638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4d82a1ce437625 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.140+0000 7f3641445700 1 -- 172.21.15.23:0/1363949956 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3638004160 con 0x7f363c07be40 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.140+0000 7f3641445700 1 -- 172.21.15.23:0/1363949956 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36380042c0 con 0x7f363c07be40 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.140+0000 7f3641445700 1 -- 172.21.15.23:0/1363949956 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36380119b0 con 0x7f363c07be40 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.141+0000 7f36446ab700 1 -- 172.21.15.23:0/1363949956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 msgr2=0x7f363c07a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:36.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.141+0000 7f36446ab700 1 --2- 172.21.15.23:0/1363949956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36380098a0 tx=0x7f3638009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.141+0000 7f36446ab700 1 -- 172.21.15.23:0/1363949956 shutdown_connections 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.141+0000 7f36446ab700 1 --2- 172.21.15.23:0/1363949956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c07a340 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.141+0000 7f36446ab700 1 -- 172.21.15.23:0/1363949956 >> 172.21.15.23:0/1363949956 conn(0x7f363c100fe0 msgr2=0x7f363c1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.142+0000 7f36446ab700 1 -- 172.21.15.23:0/1363949956 shutdown_connections 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.142+0000 7f36446ab700 1 -- 172.21.15.23:0/1363949956 wait complete. 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.143+0000 7f36446ab700 1 Processor -- start 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.143+0000 7f36446ab700 1 -- start start 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.143+0000 7f36446ab700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.143+0000 7f36446ab700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c1a01b0 con 0x7f363c07be40 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f3642447700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f3642447700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60018/0 (socket says 172.21.15.23:60018) 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f3642447700 1 -- 172.21.15.23:0/966519583 learned_addr learned my addr 172.21.15.23:0/966519583 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:36.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f3642447700 1 -- 172.21.15.23:0/966519583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3638009550 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f3642447700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3638003e60 tx=0x7f3638011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.144+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3638003a60 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.145+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3638003bc0 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.145+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36380196c0 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.145+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c107a00 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.145+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c107870 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.146+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c107b90 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.146+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f363801ab60 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.147+0000 7f36337fe700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 0x7f362803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.147+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f363804cba0 con 0x7f363c07be40 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.147+0000 7f3641c46700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 0x7f362803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:36.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.148+0000 7f3641c46700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 0x7f362803acb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f362c006fd0 tx=0x7f362c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.152+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f363801a430 con 0x7f363c07be40 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.400+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f363c1a07b0 con 0x7f363c07be40 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.401+0000 7f36337fe700 1 -- 172.21.15.23:0/966519583 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3638018b40 con 0x7f363c07be40 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 msgr2=0x7f362803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 0x7f362803acb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f362c006fd0 tx=0x7f362c006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 msgr2=0x7f363c19fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3638003e60 tx=0x7f3638011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 shutdown_connections 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3628038800 0x7f362803acb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 --2- 172.21.15.23:0/966519583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f363c07be40 0x7f363c19fc70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 >> 172.21.15.23:0/966519583 conn(0x7f363c100fe0 msgr2=0x7f363c106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:36.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 shutdown_connections 2023-11-16T23:51:36.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:36.404+0000 7f36446ab700 1 -- 172.21.15.23:0/966519583 wait complete. 2023-11-16T23:51:36.970 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T23:51:36.970 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2023-11-16T23:51:36.987 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T23:51:36.987 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2023-11-16T23:51:36.987 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2023-11-16T23:51:36.987 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T23:51:36.989 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2023-11-16T23:51:36.990 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:36.990 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2023-11-16T23:51:36.990 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2023-11-16T23:51:36.990 INFO:teuthology.orchestra.run.smithi023.stdout: Password: djra9wwv1u 2023-11-16T23:51:36.990 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:37.000 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config directory 2023-11-16T23:51:37.000 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2023-11-16T23:51:37.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:36 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/966519583' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T23:51:38.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.787+0000 7f0059dc3700 1 Processor -- start 2023-11-16T23:51:38.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.787+0000 7f0059dc3700 1 -- start start 2023-11-16T23:51:38.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.787+0000 7f0059dc3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:38.471 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.787+0000 7f0059dc3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0054106260 con 0x7f0054105910 2023-11-16T23:51:38.471 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.788+0000 7f00537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:38.471 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.788+0000 7f00537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60024/0 (socket says 172.21.15.23:60024) 2023-11-16T23:51:38.471 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.788+0000 7f00537fe700 1 -- 172.21.15.23:0/180369956 learned_addr learned my addr 172.21.15.23:0/180369956 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:38.471 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.788+0000 7f00537fe700 1 -- 172.21.15.23:0/180369956 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00541063a0 con 0x7f0054105910 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.788+0000 7f00537fe700 1 --2- 172.21.15.23:0/180369956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00440098a0 tx=0x7f0044009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacb8da08853d4df server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f00527fc700 1 -- 172.21.15.23:0/180369956 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0044004160 con 0x7f0054105910 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f00527fc700 1 -- 172.21.15.23:0/180369956 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00440042c0 con 0x7f0054105910 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f00527fc700 1 -- 172.21.15.23:0/180369956 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00440119b0 con 0x7f0054105910 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 -- 172.21.15.23:0/180369956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 msgr2=0x7f0054105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 --2- 172.21.15.23:0/180369956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00440098a0 tx=0x7f0044009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 -- 172.21.15.23:0/180369956 shutdown_connections 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 --2- 172.21.15.23:0/180369956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f0054105d20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 -- 172.21.15.23:0/180369956 >> 172.21.15.23:0/180369956 conn(0x7f0054100fa0 msgr2=0x7f00541033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 -- 172.21.15.23:0/180369956 shutdown_connections 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.789+0000 7f0059dc3700 1 -- 172.21.15.23:0/180369956 wait complete. 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.790+0000 7f0059dc3700 1 Processor -- start 2023-11-16T23:51:38.472 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f0059dc3700 1 -- start start 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f0059dc3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f0059dc3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005407c190 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f00537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f00537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60038/0 (socket says 172.21.15.23:60038) 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.791+0000 7f00537fe700 1 -- 172.21.15.23:0/2590881741 learned_addr learned my addr 172.21.15.23:0/2590881741 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f00537fe700 1 -- 172.21.15.23:0/2590881741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0044009550 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f00537fe700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0044011980 tx=0x7f0044011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0044003a60 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0044003bc0 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004401a670 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00541079a0 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.792+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0054107810 con 0x7f0054105910 2023-11-16T23:51:38.473 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.793+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0044024a00 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.793+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0054062120 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.794+0000 7f0050ff9700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 0x7f003c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.794+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f004404c950 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.794+0000 7f0052ffd700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 0x7f003c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.795+0000 7f0052ffd700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 0x7f003c03ad20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0048006fd0 tx=0x7f0048006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.799+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0044024440 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:37.947+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f00440246d0 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.007+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f005407b140 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.007+0000 7f0050ff9700 1 -- 172.21.15.23:0/2590881741 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0044018db0 con 0x7f0054105910 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 msgr2=0x7f003c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:38.474 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 0x7f003c03ad20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0048006fd0 tx=0x7f0048006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 msgr2=0x7f005407bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0044011980 tx=0x7f0044011a60 comp rx=0 tx=0).stop 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 shutdown_connections 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f003c038870 0x7f003c03ad20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.009+0000 7f0059dc3700 1 --2- 172.21.15.23:0/2590881741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0054105910 0x7f005407bc50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.010+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 >> 172.21.15.23:0/2590881741 conn(0x7f0054100fa0 msgr2=0x7f0054101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.010+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 shutdown_connections 2023-11-16T23:51:38.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.010+0000 7f0059dc3700 1 -- 172.21.15.23:0/2590881741 wait complete. 2023-11-16T23:51:39.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:38 smithi023 ceph-mon[97824]: mgrmap e13: x(active, since 6s) 2023-11-16T23:51:39.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.952+0000 7f8eb9f94700 1 Processor -- start 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.952+0000 7f8eb9f94700 1 -- start start 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.952+0000 7f8eb9f94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.952+0000 7f8eb9f94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb407a820 con 0x7f8eb407bde0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.953+0000 7f8eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.953+0000 7f8eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60042/0 (socket says 172.21.15.23:60042) 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.953+0000 7f8eb37fe700 1 -- 172.21.15.23:0/4196989985 learned_addr learned my addr 172.21.15.23:0/4196989985 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.953+0000 7f8eb37fe700 1 -- 172.21.15.23:0/4196989985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb407a960 con 0x7f8eb407bde0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.953+0000 7f8eb37fe700 1 --2- 172.21.15.23:0/4196989985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40098a0 tx=0x7f8ea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e4116b50da8bfa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.954+0000 7f8eb27fc700 1 -- 172.21.15.23:0/4196989985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ea4004160 con 0x7f8eb407bde0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.954+0000 7f8eb27fc700 1 -- 172.21.15.23:0/4196989985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea40042c0 con 0x7f8eb407bde0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.954+0000 7f8eb27fc700 1 -- 172.21.15.23:0/4196989985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ea40119b0 con 0x7f8eb407bde0 2023-11-16T23:51:39.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 -- 172.21.15.23:0/4196989985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 msgr2=0x7f8eb407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/4196989985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40098a0 tx=0x7f8ea4009bb0 comp rx=0 tx=0).stop 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 -- 172.21.15.23:0/4196989985 shutdown_connections 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/4196989985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb407a2e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 -- 172.21.15.23:0/4196989985 >> 172.21.15.23:0/4196989985 conn(0x7f8eb4100fa0 msgr2=0x7f8eb41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 -- 172.21.15.23:0/4196989985 shutdown_connections 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.955+0000 7f8eb9f94700 1 -- 172.21.15.23:0/4196989985 wait complete. 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.956+0000 7f8eb9f94700 1 Processor -- start 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb9f94700 1 -- start start 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb9f94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb9f94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb419bcf0 con 0x7f8eb407bde0 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60044/0 (socket says 172.21.15.23:60044) 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.957+0000 7f8eb37fe700 1 -- 172.21.15.23:0/3432616687 learned_addr learned my addr 172.21.15.23:0/3432616687 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb37fe700 1 -- 172.21.15.23:0/3432616687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4009550 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb37fe700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4000c00 tx=0x7f8ea4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ea40039f0 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea4003b50 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ea401a780 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb410be50 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.958+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb410bcc0 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.960+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8ea4024440 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.960+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb410bfe0 con 0x7f8eb407bde0 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.960+0000 7f8eb0ff9700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 0x7f8e9c03adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:39.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.960+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8ea404c980 con 0x7f8eb407bde0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.960+0000 7f8eb2ffd700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 0x7f8e9c03adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.961+0000 7f8eb2ffd700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 0x7f8e9c03adc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8006fd0 tx=0x7f8ea8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:38.965+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ea4018d90 con 0x7f8eb407bde0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.290+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f8eb419f3c0 con 0x7f8eb407bde0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.293+0000 7f8eb0ff9700 1 -- 172.21.15.23:0/3432616687 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8ea4018350 con 0x7f8eb407bde0 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.295+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 msgr2=0x7f8e9c03adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.295+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 0x7f8e9c03adc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8006fd0 tx=0x7f8ea8006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.295+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 msgr2=0x7f8eb419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.295+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4000c00 tx=0x7f8ea4011a30 comp rx=0 tx=0).stop 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 shutdown_connections 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8e9c038910 0x7f8e9c03adc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:39.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 --2- 172.21.15.23:0/3432616687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8eb407bde0 0x7f8eb419b7b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 >> 172.21.15.23:0/3432616687 conn(0x7f8eb4100fa0 msgr2=0x7f8eb4101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 shutdown_connections 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2023-11-16T23:51:39.296+0000 7f8eb9f94700 1 -- 172.21.15.23:0/3432616687 wait complete. 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.832 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:39.833 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2023-11-16T23:51:39.885 INFO:tasks.cephadm:Fetching config... 2023-11-16T23:51:39.885 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:51:39.885 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T23:51:39.930 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T23:51:39.931 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:51:39.931 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T23:51:40.017 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T23:51:40.017 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:51:40.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-11-16T23:51:40.128 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T23:51:40.128 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:51:40.129 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T23:51:40.216 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T23:51:40.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T23:51:40.358 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:51:40.381 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3432616687' entity='client.admin' 2023-11-16T23:51:40.392 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T23:51:40.437 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:51:40.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T23:51:40.495 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDycpkdKlRJiNlx8IdTNpzDQhVnMHFI4P3tVYht/g/lJN5QRYx8u09i9sJn1nUu6pFeINGbSANqVxv1e695JOsuot6HkQ4u2B2tcmmXC68VzcLJr9WxPoB1SxSxSoSab8RMC5LHF5UML2nIFPkvm8Srtbo4X4s4an52y0mr8Ghni+okcZz7DAjlMtnBZ0NYt9a2uYypq++EGXjLR5FDqNzgjp5nIdrc4OKF4ei1gjyWoGBQc5AusrT3kbsZDiCi65/j09xgMbn9etd4/aps3dt8AzIOch0FrhTbHyFIAbS9pqpt1bAddEs/yg5o52ZOWH/iTt6cAPHJLOqUFL7pMZ0L6+uYaufqyVzn7fH8U2SNSKJKb664qHR90URLkEzB6IZV8ZYoRUxLR69UY1fJLvxywR47A+Vn8AJjLs0222SdZE2pmtMiJtM+9ETrBTLoXDT3HWyngVV3a+PxBPakH7e7h+48fR8xs0Jr+/yMpiIJ1jG3cpbHQhR7oGNORmp3nTs= ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:51:40.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T23:51:41.579 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 -- 172.21.15.23:0/2997172377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 msgr2=0x7f523c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 --2- 172.21.15.23:0/2997172377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c072f80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f522c009910 tx=0x7f522c009c20 comp rx=0 tx=0).stop 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 -- 172.21.15.23:0/2997172377 shutdown_connections 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 --2- 172.21.15.23:0/2997172377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c072f80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 -- 172.21.15.23:0/2997172377 >> 172.21.15.23:0/2997172377 conn(0x7f523c0fb2b0 msgr2=0x7f523c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 -- 172.21.15.23:0/2997172377 shutdown_connections 2023-11-16T23:51:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.097+0000 7f524161c700 1 -- 172.21.15.23:0/2997172377 wait complete. 2023-11-16T23:51:43.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.098+0000 7f524161c700 1 Processor -- start 2023-11-16T23:51:43.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.098+0000 7f524161c700 1 -- start start 2023-11-16T23:51:43.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f524161c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:43.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f524161c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c197540 con 0x7f523c074b20 2023-11-16T23:51:43.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f523affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:43.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f523affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58354/0 (socket says 172.21.15.23:58354) 2023-11-16T23:51:43.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f523affd700 1 -- 172.21.15.23:0/3463856540 learned_addr learned my addr 172.21.15.23:0/3463856540 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:43.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f523affd700 1 -- 172.21.15.23:0/3463856540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0095f0 con 0x7f523c074b20 2023-11-16T23:51:43.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.099+0000 7f523affd700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f522c004f40 tx=0x7f522c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.100+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f522c01d070 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.100+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f522c0044e0 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.100+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c107dd0 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.100+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f522c003a20 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.100+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c107c40 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.101+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f522c003c50 con 0x7f523c074b20 2023-11-16T23:51:43.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.101+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c107f60 con 0x7f523c074b20 2023-11-16T23:51:43.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.101+0000 7f5223fff700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 0x7f522403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:43.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.102+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f522c04ca00 con 0x7f523c074b20 2023-11-16T23:51:43.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.102+0000 7f523a7fc700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 0x7f522403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:43.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.102+0000 7f523a7fc700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 0x7f522403ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5230006fd0 tx=0x7f5230006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:43.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.106+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f522c02a540 con 0x7f523c074b20 2023-11-16T23:51:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.313+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f523c190b00 con 0x7f523c074b20 2023-11-16T23:51:43.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.316+0000 7f5223fff700 1 -- 172.21.15.23:0/3463856540 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f522c02a360 con 0x7f523c074b20 2023-11-16T23:51:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 msgr2=0x7f522403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 0x7f522403ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5230006fd0 tx=0x7f5230006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 msgr2=0x7f523c197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f522c004f40 tx=0x7f522c0057c0 comp rx=0 tx=0).stop 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 shutdown_connections 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f52240388c0 0x7f522403ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 --2- 172.21.15.23:0/3463856540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f523c074b20 0x7f523c197000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 >> 172.21.15.23:0/3463856540 conn(0x7f523c0fb2b0 msgr2=0x7f523c0fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 shutdown_connections 2023-11-16T23:51:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:43.319+0000 7f524161c700 1 -- 172.21.15.23:0/3463856540 wait complete. 2023-11-16T23:51:43.769 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-16T23:51:43.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-16T23:51:44.627 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:51:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:44 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3463856540' entity='client.admin' 2023-11-16T23:51:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 -- 172.21.15.23:0/2902352762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 msgr2=0x7f31a8100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/2902352762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8100110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3198009910 tx=0x7f3198009c20 comp rx=0 tx=0).stop 2023-11-16T23:51:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 -- 172.21.15.23:0/2902352762 shutdown_connections 2023-11-16T23:51:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/2902352762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8100110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 -- 172.21.15.23:0/2902352762 >> 172.21.15.23:0/2902352762 conn(0x7f31a80fb2b0 msgr2=0x7f31a80fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 -- 172.21.15.23:0/2902352762 shutdown_connections 2023-11-16T23:51:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.130+0000 7f31ad4b0700 1 -- 172.21.15.23:0/2902352762 wait complete. 2023-11-16T23:51:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.132+0000 7f31ad4b0700 1 Processor -- start 2023-11-16T23:51:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.132+0000 7f31ad4b0700 1 -- start start 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.132+0000 7f31ad4b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.132+0000 7f31ad4b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8195340 con 0x7f31a80ffd00 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31a6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31a6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58382/0 (socket says 172.21.15.23:58382) 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31a6ffd700 1 -- 172.21.15.23:0/925537249 learned_addr learned my addr 172.21.15.23:0/925537249 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31a6ffd700 1 -- 172.21.15.23:0/925537249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31980095f0 con 0x7f31a80ffd00 2023-11-16T23:51:46.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31a6ffd700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3198004f40 tx=0x7f31980057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f319801d070 con 0x7f31a80ffd00 2023-11-16T23:51:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f319800bc10 con 0x7f31a80ffd00 2023-11-16T23:51:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a8105bd0 con 0x7f31a80ffd00 2023-11-16T23:51:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.133+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3198003a70 con 0x7f31a80ffd00 2023-11-16T23:51:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.134+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a8105a40 con 0x7f31a80ffd00 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.135+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3198003ce0 con 0x7f31a80ffd00 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.135+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8105d60 con 0x7f31a80ffd00 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.135+0000 7f318ffff700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 0x7f319003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.135+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f319804c550 con 0x7f31a80ffd00 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.135+0000 7f31a67fc700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 0x7f319003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.136+0000 7f31a67fc700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 0x7f319003ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f319c006fd0 tx=0x7f319c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.140+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f319802abf0 con 0x7f31a80ffd00 2023-11-16T23:51:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.352+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a8060f30 con 0x7f31900388c0 2023-11-16T23:51:46.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.356+0000 7f318ffff700 1 -- 172.21.15.23:0/925537249 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31a8060f30 con 0x7f31900388c0 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 msgr2=0x7f319003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 0x7f319003ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f319c006fd0 tx=0x7f319c006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 msgr2=0x7f31a8194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3198004f40 tx=0x7f31980057c0 comp rx=0 tx=0).stop 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 shutdown_connections 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900388c0 0x7f319003ad70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 --2- 172.21.15.23:0/925537249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a80ffd00 0x7f31a8194e00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:46.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 >> 172.21.15.23:0/925537249 conn(0x7f31a80fb2b0 msgr2=0x7f31a80fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:46.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 shutdown_connections 2023-11-16T23:51:46.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:46.358+0000 7f31ad4b0700 1 -- 172.21.15.23:0/925537249 wait complete. 2023-11-16T23:51:46.909 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2023-11-16T23:51:46.910 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:51:46.910 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2023-11-16T23:51:46.931 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:51:46.931 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:51:46.991 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2023-11-16T23:51:46.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi156 2023-11-16T23:51:47.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:47.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:47.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:47.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:51:47.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:51:47.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:51:47.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: Updating smithi023:/etc/ceph/ceph.conf 2023-11-16T23:51:47.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:47 smithi023 ceph-mon[97824]: Updating smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:51:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:51:48.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:48 smithi023 ceph-mon[97824]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:51:48.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:48 smithi023 ceph-mon[97824]: Updating smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:51:49.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1808952611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 msgr2=0x7fc604100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1808952611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc604100360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009910 tx=0x7fc5f4009c20 comp rx=0 tx=0).stop 2023-11-16T23:51:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1808952611 shutdown_connections 2023-11-16T23:51:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1808952611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc604100360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1808952611 >> 172.21.15.23:0/1808952611 conn(0x7fc6040fb850 msgr2=0x7fc6040fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:49.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1808952611 shutdown_connections 2023-11-16T23:51:49.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.461+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1808952611 wait complete. 2023-11-16T23:51:49.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.462+0000 7fc60a0ea700 1 Processor -- start 2023-11-16T23:51:49.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc60a0ea700 1 -- start start 2023-11-16T23:51:49.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc60a0ea700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:49.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc60a0ea700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6041978e0 con 0x7fc6040fff90 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58396/0 (socket says 172.21.15.23:58396) 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.463+0000 7fc6037fe700 1 -- 172.21.15.23:0/1390135790 learned_addr learned my addr 172.21.15.23:0/1390135790 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc6037fe700 1 -- 172.21.15.23:0/1390135790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40095f0 con 0x7fc6040fff90 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc6037fe700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4006010 tx=0x7fc5f4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:49.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f401d070 con 0x7fc6040fff90 2023-11-16T23:51:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5f400bc10 con 0x7fc6040fff90 2023-11-16T23:51:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f4003910 con 0x7fc6040fff90 2023-11-16T23:51:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60410a170 con 0x7fc6040fff90 2023-11-16T23:51:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.464+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc604109fe0 con 0x7fc6040fff90 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.465+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc5f400bd80 con 0x7fc6040fff90 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.465+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60410a300 con 0x7fc6040fff90 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.466+0000 7fc600ff9700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 0x7fc5ec03adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.466+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5f404c5a0 con 0x7fc6040fff90 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.466+0000 7fc602ffd700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 0x7fc5ec03adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.467+0000 7fc602ffd700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 0x7fc5ec03adc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8006fd0 tx=0x7fc5f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.471+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5f4029360 con 0x7fc6040fff90 2023-11-16T23:51:49.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:49.681+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) v1 -- 0x7fc604197ee0 con 0x7fc5ec038910 2023-11-16T23:51:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:51.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:50 smithi023 ceph-mon[97824]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:51:51.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:50 smithi023 ceph-mon[97824]: Deploying cephadm binary to smithi156 2023-11-16T23:51:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.877+0000 7fc600ff9700 1 -- 172.21.15.23:0/1390135790 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc604197ee0 con 0x7fc5ec038910 2023-11-16T23:51:51.880 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi156' with addr '172.21.15.156' 2023-11-16T23:51:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 msgr2=0x7fc5ec03adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 0x7fc5ec03adc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8006fd0 tx=0x7fc5f8006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 msgr2=0x7fc6041973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4006010 tx=0x7fc5f4004ff0 comp rx=0 tx=0).stop 2023-11-16T23:51:51.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 shutdown_connections 2023-11-16T23:51:51.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5ec038910 0x7fc5ec03adc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 --2- 172.21.15.23:0/1390135790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6040fff90 0x7fc6041973a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 >> 172.21.15.23:0/1390135790 conn(0x7fc6040fb850 msgr2=0x7fc6041038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 shutdown_connections 2023-11-16T23:51:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:51.880+0000 7fc60a0ea700 1 -- 172.21.15.23:0/1390135790 wait complete. 2023-11-16T23:51:52.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:51 smithi023 ceph-mon[97824]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:51:52.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:51 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:52.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:51 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:51:52.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T23:51:53.225 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:51:53.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:52 smithi023 ceph-mon[97824]: Added host smithi156 2023-11-16T23:51:53.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:54.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:54 smithi023 ceph-mon[97824]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:51:57.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 -- 172.21.15.23:0/3944831619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 msgr2=0x7fcbac100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/3944831619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac100360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009910 tx=0x7fcb9c009c20 comp rx=0 tx=0).stop 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 -- 172.21.15.23:0/3944831619 shutdown_connections 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/3944831619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac100360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 -- 172.21.15.23:0/3944831619 >> 172.21.15.23:0/3944831619 conn(0x7fcbac0fb850 msgr2=0x7fcbac0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 -- 172.21.15.23:0/3944831619 shutdown_connections 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.733+0000 7fcbb1f41700 1 -- 172.21.15.23:0/3944831619 wait complete. 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.734+0000 7fcbb1f41700 1 Processor -- start 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbb1f41700 1 -- start start 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbb1f41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbb1f41700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac074d70 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbab7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbab7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41536/0 (socket says 172.21.15.23:41536) 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.735+0000 7fcbab7fe700 1 -- 172.21.15.23:0/2460092260 learned_addr learned my addr 172.21.15.23:0/2460092260 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcbab7fe700 1 -- 172.21.15.23:0/2460092260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0095f0 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcbab7fe700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c006010 tx=0x7fcb9c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb9c01d070 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb9c00bc10 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb9c003910 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac10a170 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.736+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac109fe0 con 0x7fcbac0fff90 2023-11-16T23:51:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.737+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac10a300 con 0x7fcbac0fff90 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.737+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fcb9c00bd80 con 0x7fcbac0fff90 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.738+0000 7fcba8ff9700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 0x7fcb9403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.738+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcb9c04c5a0 con 0x7fcbac0fff90 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.738+0000 7fcbaaffd700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 0x7fcb9403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.739+0000 7fcbaaffd700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 0x7fcb9403ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcba0006fd0 tx=0x7fcba0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.743+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcb9c029360 con 0x7fcbac0fff90 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.954+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcbac074320 con 0x7fcb940388c0 2023-11-16T23:51:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.955+0000 7fcba8ff9700 1 -- 172.21.15.23:0/2460092260 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fcbac074320 con 0x7fcb940388c0 2023-11-16T23:51:57.889 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:51:57.889 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2023-11-16T23:51:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 msgr2=0x7fcb9403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 0x7fcb9403ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcba0006fd0 tx=0x7fcba0006e40 comp rx=0 tx=0).stop 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 msgr2=0x7fcbac072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c006010 tx=0x7fcb9c004ff0 comp rx=0 tx=0).stop 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 shutdown_connections 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb940388c0 0x7fcb9403ad70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 --2- 172.21.15.23:0/2460092260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbac0fff90 0x7fcbac072df0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:51:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 >> 172.21.15.23:0/2460092260 conn(0x7fcbac0fb850 msgr2=0x7fcbac104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:51:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 shutdown_connections 2023-11-16T23:51:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:51:54.958+0000 7fcbb1f41700 1 -- 172.21.15.23:0/2460092260 wait complete. 2023-11-16T23:51:57.928 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2023-11-16T23:51:57.928 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:51:57.928 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2023-11-16T23:51:57.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:55 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:57.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:55 smithi023 ceph-mon[97824]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:51:57.929 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:55 smithi023 ceph-mon[97824]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:51:57.950 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:51:57.950 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:51:58.010 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2023-11-16T23:51:58.010 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi173 2023-11-16T23:51:58.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:51:58.876 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:51:59.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:59 smithi023 ceph-mon[97824]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:51:59.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:59 smithi023 ceph-mon[97824]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:51:59.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:51:59 smithi023 ceph-mon[97824]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 -- 172.21.15.23:0/83398375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 msgr2=0x7f14bc100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 --2- 172.21.15.23:0/83398375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc100360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009910 tx=0x7f14ac009c20 comp rx=0 tx=0).stop 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 -- 172.21.15.23:0/83398375 shutdown_connections 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 --2- 172.21.15.23:0/83398375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc100360 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 -- 172.21.15.23:0/83398375 >> 172.21.15.23:0/83398375 conn(0x7f14bc0fb850 msgr2=0x7f14bc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 -- 172.21.15.23:0/83398375 shutdown_connections 2023-11-16T23:52:00.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.368+0000 7f14c25fc700 1 -- 172.21.15.23:0/83398375 wait complete. 2023-11-16T23:52:00.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.369+0000 7f14c25fc700 1 Processor -- start 2023-11-16T23:52:00.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.369+0000 7f14c25fc700 1 -- start start 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14c25fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14c25fc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc074d70 con 0x7f14bc0fff90 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41552/0 (socket says 172.21.15.23:41552) 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14bbfff700 1 -- 172.21.15.23:0/2650833620 learned_addr learned my addr 172.21.15.23:0/2650833620 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:52:00.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14bbfff700 1 -- 172.21.15.23:0/2650833620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0095f0 con 0x7f14bc0fff90 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.370+0000 7f14bbfff700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006010 tx=0x7f14ac004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.371+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac01d070 con 0x7f14bc0fff90 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.371+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14ac00bc10 con 0x7f14bc0fff90 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.371+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc10a170 con 0x7f14bc0fff90 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.371+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac003910 con 0x7f14bc0fff90 2023-11-16T23:52:00.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.371+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc109fe0 con 0x7f14bc0fff90 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.372+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f14ac00bd80 con 0x7f14bc0fff90 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.372+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc10a300 con 0x7f14bc0fff90 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.372+0000 7f14b97fa700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 0x7f14a403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.372+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f14ac04c560 con 0x7f14bc0fff90 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.373+0000 7f14bb7fe700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 0x7f14a403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:00.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.373+0000 7f14bb7fe700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 0x7f14a403ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006fd0 tx=0x7f14b0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:00.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.378+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f14ac029360 con 0x7f14bc0fff90 2023-11-16T23:52:00.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:00 smithi023 ceph-mon[97824]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:00.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:00.589+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7f14bc074320 con 0x7f14a40388c0 2023-11-16T23:52:02.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:02 smithi023 ceph-mon[97824]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:02 smithi023 ceph-mon[97824]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:02 smithi023 ceph-mon[97824]: Deploying cephadm binary to smithi173 2023-11-16T23:52:02.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.718+0000 7f14b97fa700 1 -- 172.21.15.23:0/2650833620 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f14bc074320 con 0x7f14a40388c0 2023-11-16T23:52:02.720 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi173' with addr '172.21.15.173' 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 msgr2=0x7f14a403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 0x7f14a403ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006fd0 tx=0x7f14b0006e40 comp rx=0 tx=0).stop 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 msgr2=0x7f14bc072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006010 tx=0x7f14ac004ff0 comp rx=0 tx=0).stop 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 shutdown_connections 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14a40388c0 0x7f14a403ad70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 --2- 172.21.15.23:0/2650833620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc0fff90 0x7f14bc072df0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 >> 172.21.15.23:0/2650833620 conn(0x7f14bc0fb850 msgr2=0x7f14bc104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:02.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 shutdown_connections 2023-11-16T23:52:02.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:02.721+0000 7f14c25fc700 1 -- 172.21.15.23:0/2650833620 wait complete. 2023-11-16T23:52:03.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T23:52:04.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:03 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:04.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:03 smithi023 ceph-mon[97824]: Added host smithi173 2023-11-16T23:52:04.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:03 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:04.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:03 smithi023 ceph-mon[97824]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:04.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:03 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:04.021 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:52:05.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 -- 172.21.15.23:0/2976654561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 msgr2=0x7f97a0100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 --2- 172.21.15.23:0/2976654561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a0100360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9790009910 tx=0x7f9790009c20 comp rx=0 tx=0).stop 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 -- 172.21.15.23:0/2976654561 shutdown_connections 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 --2- 172.21.15.23:0/2976654561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a0100360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 -- 172.21.15.23:0/2976654561 >> 172.21.15.23:0/2976654561 conn(0x7f97a00fb850 msgr2=0x7f97a00fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 -- 172.21.15.23:0/2976654561 shutdown_connections 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.543+0000 7f97a5559700 1 -- 172.21.15.23:0/2976654561 wait complete. 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.544+0000 7f97a5559700 1 Processor -- start 2023-11-16T23:52:05.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.544+0000 7f97a5559700 1 -- start start 2023-11-16T23:52:05.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.545+0000 7f97a5559700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.545+0000 7f97a5559700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a0195700 con 0x7f97a00fff90 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.545+0000 7f979effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.545+0000 7f979effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37530/0 (socket says 172.21.15.23:37530) 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.545+0000 7f979effd700 1 -- 172.21.15.23:0/153903637 learned_addr learned my addr 172.21.15.23:0/153903637 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f979effd700 1 -- 172.21.15.23:0/153903637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900095f0 con 0x7f97a00fff90 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f979effd700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9790004f40 tx=0x7f9790005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f979001d070 con 0x7f97a00fff90 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9790004370 con 0x7f97a00fff90 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97900039b0 con 0x7f97a00fff90 2023-11-16T23:52:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a010a170 con 0x7f97a00fff90 2023-11-16T23:52:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.546+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a0109fe0 con 0x7f97a00fff90 2023-11-16T23:52:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.547+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9790003c20 con 0x7f97a00fff90 2023-11-16T23:52:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.547+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a010a300 con 0x7f97a00fff90 2023-11-16T23:52:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.548+0000 7f9787fff700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 0x7f978803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.548+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f979004c5b0 con 0x7f97a00fff90 2023-11-16T23:52:05.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.548+0000 7f979e7fc700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 0x7f978803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:05.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.549+0000 7f979e7fc700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 0x7f978803ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9794006fd0 tx=0x7f9794006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:05.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.553+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f979002a980 con 0x7f97a00fff90 2023-11-16T23:52:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.766+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97a01080d0 con 0x7f97880388c0 2023-11-16T23:52:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.767+0000 7f9787fff700 1 -- 172.21.15.23:0/153903637 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f97a01080d0 con 0x7f97880388c0 2023-11-16T23:52:05.769 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:52:05.769 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2023-11-16T23:52:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.769+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 msgr2=0x7f978803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.769+0000 7f97a5559700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 0x7f978803ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9794006fd0 tx=0x7f9794006e40 comp rx=0 tx=0).stop 2023-11-16T23:52:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 msgr2=0x7f97a01951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9790004f40 tx=0x7f9790005020 comp rx=0 tx=0).stop 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 shutdown_connections 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97880388c0 0x7f978803ad70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 --2- 172.21.15.23:0/153903637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97a00fff90 0x7f97a01951c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 >> 172.21.15.23:0/153903637 conn(0x7f97a00fb850 msgr2=0x7f97a0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 shutdown_connections 2023-11-16T23:52:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:05.770+0000 7f97a5559700 1 -- 172.21.15.23:0/153903637 wait complete. 2023-11-16T23:52:06.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:06 smithi023 ceph-mon[97824]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:06.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:06.305 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T23:52:06.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T23:52:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:52:07.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:07 smithi023 ceph-mon[97824]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:52:08.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 -- 172.21.15.23:0/859164329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 msgr2=0x7feef0100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:08.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 --2- 172.21.15.23:0/859164329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0100360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feeec009910 tx=0x7feeec009c20 comp rx=0 tx=0).stop 2023-11-16T23:52:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 -- 172.21.15.23:0/859164329 shutdown_connections 2023-11-16T23:52:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 --2- 172.21.15.23:0/859164329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0100360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 -- 172.21.15.23:0/859164329 >> 172.21.15.23:0/859164329 conn(0x7feef00fb850 msgr2=0x7feef00fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 -- 172.21.15.23:0/859164329 shutdown_connections 2023-11-16T23:52:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.637+0000 7feef87cc700 1 -- 172.21.15.23:0/859164329 wait complete. 2023-11-16T23:52:08.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.638+0000 7feef87cc700 1 Processor -- start 2023-11-16T23:52:08.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.638+0000 7feef87cc700 1 -- start start 2023-11-16T23:52:08.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef87cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:08.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef87cc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0074d70 con 0x7feef00fff90 2023-11-16T23:52:08.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef6568700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:08.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef6568700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37546/0 (socket says 172.21.15.23:37546) 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef6568700 1 -- 172.21.15.23:0/4249602981 learned_addr learned my addr 172.21.15.23:0/4249602981 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.639+0000 7feef6568700 1 -- 172.21.15.23:0/4249602981 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec0095f0 con 0x7feef00fff90 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feef6568700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feeec004f40 tx=0x7feeec005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feeec01d070 con 0x7feef00fff90 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeec004370 con 0x7feef00fff90 2023-11-16T23:52:08.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef010a170 con 0x7feef00fff90 2023-11-16T23:52:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feeec0039b0 con 0x7feef00fff90 2023-11-16T23:52:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.640+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef0109fe0 con 0x7feef00fff90 2023-11-16T23:52:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.641+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef010a300 con 0x7feef00fff90 2023-11-16T23:52:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.641+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7feeec003c20 con 0x7feef00fff90 2023-11-16T23:52:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.642+0000 7feee77fe700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 0x7feedc03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.642+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feeec04c810 con 0x7feef00fff90 2023-11-16T23:52:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.642+0000 7feef5d67700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 0x7feedc03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.643+0000 7feef5d67700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 0x7feedc03ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feee0006fd0 tx=0x7feee0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.647+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feeec029360 con 0x7feef00fff90 2023-11-16T23:52:08.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:08.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:08.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:08.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:08.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:08.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:52:08.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:08.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:08.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:08.858+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7feef0108210 con 0x7feef00fff90 2023-11-16T23:52:09.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.304+0000 7feee77fe700 1 -- 172.21.15.23:0/4249602981 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7feeec026020 con 0x7feef00fff90 2023-11-16T23:52:09.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 msgr2=0x7feedc03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 0x7feedc03ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feee0006fd0 tx=0x7feee0006e40 comp rx=0 tx=0).stop 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 msgr2=0x7feef0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feeec004f40 tx=0x7feeec005020 comp rx=0 tx=0).stop 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 shutdown_connections 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7feedc0388c0 0x7feedc03ad70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 --2- 172.21.15.23:0/4249602981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feef00fff90 0x7feef0072df0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.306+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 >> 172.21.15.23:0/4249602981 conn(0x7feef00fb850 msgr2=0x7feef0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.307+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 shutdown_connections 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:09.307+0000 7feef87cc700 1 -- 172.21.15.23:0/4249602981 wait complete. 2023-11-16T23:52:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2023-11-16T23:52:09.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:09.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:09.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:09.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T23:52:09.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:09.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:09.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:09.745 INFO:tasks.cephadm:Adding mon.a on smithi023 2023-11-16T23:52:09.746 INFO:tasks.cephadm:Adding mon.b on smithi156 2023-11-16T23:52:09.746 INFO:tasks.cephadm:Adding mon.c on smithi173 2023-11-16T23:52:09.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c' 2023-11-16T23:52:10.414 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:10 smithi023 ceph-mon[97824]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:10.907 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:10 smithi023 ceph-mon[97824]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:11.558 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:10 smithi023 ceph-mon[97824]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:11.558 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:10 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T23:52:11.558 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:10 smithi023 ceph-mon[97824]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:11.620 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 -- 172.21.15.173:0/3802477235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 msgr2=0x7fdfa4100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/3802477235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa4100360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009910 tx=0x7fdf98009c20 comp rx=0 tx=0).stop 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 -- 172.21.15.173:0/3802477235 shutdown_connections 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/3802477235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa4100360 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 -- 172.21.15.173:0/3802477235 >> 172.21.15.173:0/3802477235 conn(0x7fdfa40fb850 msgr2=0x7fdfa40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 -- 172.21.15.173:0/3802477235 shutdown_connections 2023-11-16T23:52:11.621 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 -- 172.21.15.173:0/3802477235 wait complete. 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.619+0000 7fdfa9e75700 1 Processor -- start 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa9e75700 1 -- start start 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa9e75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa9e75700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4195700 con 0x7fdfa40fff90 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.173:56726/0 (socket says 172.21.15.173:56726) 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.620+0000 7fdfa37fe700 1 -- 172.21.15.173:0/2677077822 learned_addr learned my addr 172.21.15.173:0/2677077822 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:52:11.622 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa37fe700 1 -- 172.21.15.173:0/2677077822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980095f0 con 0x7fdfa40fff90 2023-11-16T23:52:11.623 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa37fe700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdf98004f40 tx=0x7fdf98005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf9801d070 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa410a170 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf98004370 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf980039b0 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.621+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa4109fe0 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.622+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fdf98003c70 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.622+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa410a300 con 0x7fdfa40fff90 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.622+0000 7fdfa0ff9700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 0x7fdf8c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:11.624 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.622+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf9804c8a0 con 0x7fdfa40fff90 2023-11-16T23:52:11.625 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.623+0000 7fdfa2ffd700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 0x7fdf8c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:11.625 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.623+0000 7fdfa2ffd700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 0x7fdf8c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf94006fd0 tx=0x7fdf94006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:11.635 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.628+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdf98029360 con 0x7fdfa40fff90 2023-11-16T23:52:11.830 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.823+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfa4108190 con 0x7fdf8c038870 2023-11-16T23:52:11.832 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.831+0000 7fdfa0ff9700 1 -- 172.21.15.173:0/2677077822 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdfa4108190 con 0x7fdf8c038870 2023-11-16T23:52:11.832 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2023-11-16T23:52:11.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 msgr2=0x7fdf8c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:11.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 0x7fdf8c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf94006fd0 tx=0x7fdf94006e40 comp rx=0 tx=0).stop 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 msgr2=0x7fdfa41951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdf98004f40 tx=0x7fdf98005020 comp rx=0 tx=0).stop 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 shutdown_connections 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdf8c038870 0x7fdf8c03ad20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 --2- 172.21.15.173:0/2677077822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdfa40fff90 0x7fdfa41951c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 >> 172.21.15.173:0/2677077822 conn(0x7fdfa40fb850 msgr2=0x7fdfa4104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 shutdown_connections 2023-11-16T23:52:11.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:11.832+0000 7fdfa9e75700 1 -- 172.21.15.173:0/2677077822 wait complete. 2023-11-16T23:52:12.268 DEBUG:teuthology.orchestra.run.smithi156:mon.b> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.b.service 2023-11-16T23:52:12.271 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.c.service 2023-11-16T23:52:12.273 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T23:52:12.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T23:52:12.302 INFO:journalctl@ceph.mon.c.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:52:12.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:52:12.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:13.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:13 smithi023 ceph-mon[97824]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:13.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:13 smithi023 ceph-mon[97824]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c;count:3 2023-11-16T23:52:13.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:13 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:13.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:13 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:13.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:13 smithi023 ceph-mon[97824]: Deploying daemon mon.c on smithi173 2023-11-16T23:52:14.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:14 smithi023 ceph-mon[97824]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 -- 172.21.15.173:0/3211314218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 msgr2=0x7f8e04100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:15.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 --2- 172.21.15.173:0/3211314218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e04100250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009910 tx=0x7f8e00009c20 comp rx=0 tx=0).stop 2023-11-16T23:52:15.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 -- 172.21.15.173:0/3211314218 shutdown_connections 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 --2- 172.21.15.173:0/3211314218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e04100250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 -- 172.21.15.173:0/3211314218 >> 172.21.15.173:0/3211314218 conn(0x7f8e04074350 msgr2=0x7f8e04074750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 -- 172.21.15.173:0/3211314218 shutdown_connections 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 -- 172.21.15.173:0/3211314218 wait complete. 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.069+0000 7f8e0be62700 1 Processor -- start 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e0be62700 1 -- start start 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e0be62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:15.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e0be62700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e04103770 con 0x7f8e040ffe80 2023-11-16T23:52:15.073 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e09bfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:15.073 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e09bfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.173:56732/0 (socket says 172.21.15.173:56732) 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.070+0000 7f8e09bfe700 1 -- 172.21.15.173:0/727910820 learned_addr learned my addr 172.21.15.173:0/727910820 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8e09bfe700 1 -- 172.21.15.173:0/727910820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000095f0 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8e09bfe700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8e00006010 tx=0x7f8e00004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e0001d070 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e0000bc10 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e04109090 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e00003910 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.071+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e04108f00 con 0x7f8e040ffe80 2023-11-16T23:52:15.074 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.072+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e04109220 con 0x7f8e040ffe80 2023-11-16T23:52:15.075 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.073+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8e00003bd0 con 0x7f8e040ffe80 2023-11-16T23:52:15.075 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.073+0000 7f8dfaffd700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 0x7f8df003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:15.075 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.073+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8e0004c4e0 con 0x7f8e040ffe80 2023-11-16T23:52:15.075 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.073+0000 7f8e093fd700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 0x7f8df003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:15.075 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.074+0000 7f8e093fd700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 0x7f8df003ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8df4006fd0 tx=0x7f8df4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:15.080 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.078+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8e0000ba70 con 0x7f8e040ffe80 2023-11-16T23:52:15.355 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.354+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e041024f0 con 0x7f8e040ffe80 2023-11-16T23:52:15.356 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.354+0000 7f8dfaffd700 1 -- 172.21.15.173:0/727910820 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f8e00026070 con 0x7f8e040ffe80 2023-11-16T23:52:15.357 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:52:15.357 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T23:50:29.183152Z","created":"2023-11-16T23:50:29.183152Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 msgr2=0x7f8df003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 0x7f8df003ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8df4006fd0 tx=0x7f8df4006e40 comp rx=0 tx=0).stop 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 msgr2=0x7f8e041017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8e00006010 tx=0x7f8e00004ff0 comp rx=0 tx=0).stop 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 shutdown_connections 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8df0038870 0x7f8df003ad20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:15.359 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 --2- 172.21.15.173:0/727910820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e040ffe80 0x7f8e041017f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:15.360 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 >> 172.21.15.173:0/727910820 conn(0x7f8e04074350 msgr2=0x7f8e040fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:15.360 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 shutdown_connections 2023-11-16T23:52:15.360 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:15.357+0000 7f8e0be62700 1 -- 172.21.15.173:0/727910820 wait complete. 2023-11-16T23:52:15.360 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2023-11-16T23:52:15.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 podman[131478]: 2023-11-16T23:52:15.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 bash[131478]: 55a30f0f87ef27cb49f1c3c2587237421c1223b6ce0b703e5905c18daf405016 2023-11-16T23:52:15.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:15 smithi023 ceph-mon[97824]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev), process ceph-mon, pid 7 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pidfile_write: ignore empty --pid-file 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: load: jerasure load: lrc 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: RocksDB version: 7.9.2 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Git sha 0 2023-11-16T23:52:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Compile date 2023-11-16 19:40:53 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: DB SUMMARY 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: DB Session ID: O7NXE67R8GEADLICENOO 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: CURRENT file: CURRENT 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.error_if_exists: 0 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.create_if_missing: 0 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.paranoid_checks: 1 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T23:52:15.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.env: 0x563e6183c720 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.info_log: 0x563e640f33c0 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.statistics: (nil) 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.use_fsync: 0 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_log_file_size: 0 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_fallocate: 1 2023-11-16T23:52:15.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.use_direct_reads: 0 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.db_log_dir: 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.wal_dir: 2023-11-16T23:52:15.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T23:52:15.810 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.write_buffer_manager: 0x563e633643c0 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T23:52:15.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.unordered_write: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.row_cache: None 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.wal_filter: None 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T23:52:15.812 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.two_write_queues: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.wal_compression: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.atomic_flush: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.log_readahead_size: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T23:52:15.813 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_background_jobs: 2 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_background_compactions: -1 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_subcompactions: 1 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T23:52:15.814 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_open_files: -1 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_background_flushes: -1 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Compression algorithms supported: 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kZSTD supported: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kXpressCompression supported: 0 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kZlibCompression supported: 1 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kSnappyCompression supported: 1 2023-11-16T23:52:15.815 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kLZ4Compression supported: 1 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: kBZip2Compression supported: 0 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.merge_operator: 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_filter: None 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T23:52:15.816 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563e640f34e0) 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks: 1 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_top_level_index_and_filter: 1 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_type: 0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_index_type: 0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_shortening: 1 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: checksum: 4 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: no_block_cache: 0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache: 0x563e634051f0 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_name: BinnedLRUCache 2023-11-16T23:52:15.817 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_options: 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: capacity : 536870912 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_shard_bits : 4 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: strict_capacity_limit : 0 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: high_pri_pool_ratio: 0.000 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_compressed: (nil) 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: persistent_cache: (nil) 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size: 4096 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size_deviation: 10 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_restart_interval: 16 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_block_restart_interval: 1 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: metadata_block_size: 4096 2023-11-16T23:52:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout: partition_filters: 0 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: use_delta_encoding: 1 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: filter_policy: bloomfilter 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: whole_key_filtering: 1 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: verify_compression: 0 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: read_amp_bytes_per_bit: 0 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: format_version: 5 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_index_compression: 1 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_align: 0 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: max_auto_readahead_size: 262144 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: prepopulate_block_cache: 0 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: initial_auto_readahead_size: 8192 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T23:52:15.819 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression: NoCompression 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.num_levels: 7 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T23:52:15.820 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T23:52:15.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T23:52:15.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T23:52:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.table_properties_collectors: 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.inplace_update_support: 0 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T23:52:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.bloom_locality: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.max_successive_merges: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.ttl: 2592000 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enable_blob_files: false 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.min_blob_size: 0 2023-11-16T23:52:15.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1c1afa3e-f9ec-40ef-93e9-2470144036f8 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178735487687, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T23:52:15.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178735488096, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700178735, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1c1afa3e-f9ec-40ef-93e9-2470144036f8", "db_session_id": "O7NXE67R8GEADLICENOO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178735488164, "job": 1, "event": "recovery_finished"} 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563e634a2000 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: DB pointer 0x563e6348e000 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** DB Stats ** 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:15.827 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2023-11-16T23:52:15.828 INFO:journalctl@ceph.mon.c.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:52:15.829 INFO:journalctl@ceph.mon.c.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache BinnedLRUCache@0x563e634051f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: starting mon.c rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(???) e0 preinit fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).mds e1 new map 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).mds e1 print_map 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: e1 2023-11-16T23:52:15.830 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout: legacy client fscid: -1 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout: No filesystems configured 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:15.831 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Deploying cephadm binary to smithi173 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Added host smithi173 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:15.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.835 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c;count:3 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: Deploying daemon mon.c on smithi173 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:15.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T23:52:15.837 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 systemd[1]: Started Ceph mon.c for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:52:15.837 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:15 smithi173 ceph-mon[131534]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-11-16T23:52:16.874 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T23:52:16.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T23:52:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:52:20.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:20.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: mon.a calling monitor election 2023-11-16T23:52:20.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: mon.c calling monitor election 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: fsmap 2023-11-16T23:52:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: mgrmap e13: x(active, since 49s) 2023-11-16T23:52:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: overall HEALTH_OK 2023-11-16T23:52:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: mon.a calling monitor election 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: mon.c calling monitor election 2023-11-16T23:52:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: fsmap 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: mgrmap e13: x(active, since 49s) 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: overall HEALTH_OK 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 -- 172.21.15.173:0/3701448075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 msgr2=0x7fbe9c005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/3701448075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbe9c005a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00fec90 tx=0x7fbea000feb0 comp rx=0 tx=0).stop 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 -- 172.21.15.173:0/3701448075 shutdown_connections 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/3701448075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbe9c005a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 -- 172.21.15.173:0/3701448075 >> 172.21.15.173:0/3701448075 conn(0x7fbeb00fa8a0 msgr2=0x7fbeb00fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 -- 172.21.15.173:0/3701448075 shutdown_connections 2023-11-16T23:52:21.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.773+0000 7fbeb7b02700 1 -- 172.21.15.173:0/3701448075 wait complete. 2023-11-16T23:52:21.775 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.774+0000 7fbeb7b02700 1 Processor -- start 2023-11-16T23:52:21.775 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.774+0000 7fbeb7b02700 1 -- start start 2023-11-16T23:52:21.816 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb7b02700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbeb01009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:21.816 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb7b02700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:21.816 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb7b02700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0102950 con 0x7fbe9c0035c0 2023-11-16T23:52:21.817 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb7b02700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb01a14d0 con 0x7fbeb0100f10 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51050/0 (socket says 172.21.15.173:51050) 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 -- 172.21.15.173:0/496705776 learned_addr learned my addr 172.21.15.173:0/496705776 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 -- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 msgr2=0x7fbeb0101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 -- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 msgr2=0x7fbeb0101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb509d700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.775+0000 7fbeb589e700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbeb01009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb589e700 1 -- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 msgr2=0x7fbeb0101380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb589e700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:21.819 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb589e700 1 -- 172.21.15.173:0/496705776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea000f900 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb589e700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbeb01009d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbea0011510 tx=0x7fbea0005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbea0013b00 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbea00044f0 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb0106bb0 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbea0003b50 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.776+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb0106a20 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.777+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbea0011590 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.777+0000 7fbea6ffd700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 0x7fbe9403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.777+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb0106d40 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.778+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbea004dd80 con 0x7fbe9c0035c0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.778+0000 7fbeb509d700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 0x7fbe9403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.779+0000 7fbeb509d700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 0x7fbe9403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009180 tx=0x7fbeac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:21.820 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:21.784+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbea0010810 con 0x7fbe9c0035c0 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: Deploying daemon mon.b on smithi156 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:21 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:22.050 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.050+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbeb0062120 con 0x7fbe9c0035c0 2023-11-16T23:52:22.051 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.050+0000 7fbea6ffd700 1 -- 172.21.15.173:0/496705776 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fbea000b650 con 0x7fbe9c0035c0 2023-11-16T23:52:22.051 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:52:22.052 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T23:52:15.512619Z","created":"2023-11-16T23:50:29.183152Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 msgr2=0x7fbe9403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 0x7fbe9403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009180 tx=0x7fbeac008040 comp rx=0 tx=0).stop 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 msgr2=0x7fbeb01009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbeb01009d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbea0011510 tx=0x7fbea0005250 comp rx=0 tx=0).stop 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 shutdown_connections 2023-11-16T23:52:22.053 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c0035c0 0x7fbeb01009d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe940386b0 0x7fbe9403ab60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 --2- 172.21.15.173:0/496705776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbeb0100f10 0x7fbeb0101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 >> 172.21.15.173:0/496705776 conn(0x7fbeb00fa8a0 msgr2=0x7fbeb0069370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 shutdown_connections 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:22.052+0000 7fbeb7b02700 1 -- 172.21.15.173:0/496705776 wait complete. 2023-11-16T23:52:22.054 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2023-11-16T23:52:22.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:22.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:22.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:22.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: Deploying daemon mon.b on smithi156 2023-11-16T23:52:22.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:22.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:21 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/496705776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:22.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:52:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:52:22.512+0000 7f3b10c2d700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-16T23:52:22.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:22 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/496705776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:23.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 systemd[1]: Starting Ceph mon.b for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:52:23.489 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T23:52:23.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 podman[131066]: 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 bash[131066]: d62b23c6552d48b2e3df90f039382f31618e757a2f4d5b1fb7968d9fef5c2d92 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev), process ceph-mon, pid 7 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pidfile_write: ignore empty --pid-file 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: load: jerasure load: lrc 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: RocksDB version: 7.9.2 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Git sha 0 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Compile date 2023-11-16 19:40:53 2023-11-16T23:52:23.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: DB SUMMARY 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: DB Session ID: C1A3P9HQQB3JB3PUJ40C 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: CURRENT file: CURRENT 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.error_if_exists: 0 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.create_if_missing: 0 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.paranoid_checks: 1 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T23:52:23.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.env: 0x560646823720 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.info_log: 0x560649a153c0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.statistics: (nil) 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.use_fsync: 0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_log_file_size: 0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_fallocate: 1 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T23:52:23.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.use_direct_reads: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.db_log_dir: 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.wal_dir: 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T23:52:23.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.write_buffer_manager: 0x560648c863c0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.unordered_write: 0 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T23:52:23.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.row_cache: None 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.wal_filter: None 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.two_write_queues: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.wal_compression: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.atomic_flush: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.log_readahead_size: 0 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T23:52:23.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T23:52:23.731 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T23:52:23.731 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T23:52:23.732 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T23:52:23.732 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T23:52:23.732 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T23:52:23.732 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_background_jobs: 2 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_background_compactions: -1 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_subcompactions: 1 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T23:52:23.733 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T23:52:23.734 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T23:52:23.734 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T23:52:23.734 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T23:52:23.734 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_open_files: -1 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_background_flushes: -1 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Compression algorithms supported: 2023-11-16T23:52:23.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kZSTD supported: 0 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kXpressCompression supported: 0 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kZlibCompression supported: 1 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kSnappyCompression supported: 1 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kLZ4Compression supported: 1 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: kBZip2Compression supported: 0 2023-11-16T23:52:23.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T23:52:23.739 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T23:52:23.739 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-11-16T23:52:23.739 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T23:52:23.739 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T23:52:23.739 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.merge_operator: 2023-11-16T23:52:23.740 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_filter: None 2023-11-16T23:52:23.740 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T23:52:23.740 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T23:52:23.740 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T23:52:23.740 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T23:52:23.741 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560649a154e0) 2023-11-16T23:52:23.741 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks: 1 2023-11-16T23:52:23.741 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T23:52:23.741 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T23:52:23.741 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_top_level_index_and_filter: 1 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_type: 0 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_index_type: 0 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_shortening: 1 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: checksum: 4 2023-11-16T23:52:23.742 INFO:journalctl@ceph.mon.b.smithi156.stdout: no_block_cache: 0 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache: 0x560648d271f0 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_name: BinnedLRUCache 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_options: 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: capacity : 536870912 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_shard_bits : 4 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: strict_capacity_limit : 0 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: high_pri_pool_ratio: 0.000 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_compressed: (nil) 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: persistent_cache: (nil) 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size: 4096 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size_deviation: 10 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_restart_interval: 16 2023-11-16T23:52:23.743 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_block_restart_interval: 1 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: metadata_block_size: 4096 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: partition_filters: 0 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_delta_encoding: 1 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: filter_policy: bloomfilter 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: whole_key_filtering: 1 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: verify_compression: 0 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: read_amp_bytes_per_bit: 0 2023-11-16T23:52:23.744 INFO:journalctl@ceph.mon.b.smithi156.stdout: format_version: 5 2023-11-16T23:52:23.745 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_index_compression: 1 2023-11-16T23:52:23.745 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_align: 0 2023-11-16T23:52:23.745 INFO:journalctl@ceph.mon.b.smithi156.stdout: max_auto_readahead_size: 262144 2023-11-16T23:52:23.745 INFO:journalctl@ceph.mon.b.smithi156.stdout: prepopulate_block_cache: 0 2023-11-16T23:52:23.759 INFO:journalctl@ceph.mon.b.smithi156.stdout: initial_auto_readahead_size: 8192 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression: NoCompression 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T23:52:23.760 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.num_levels: 7 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T23:52:23.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T23:52:23.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T23:52:23.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T23:52:23.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T23:52:23.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:52:23.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T23:52:23.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T23:52:23.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:52:23.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:52:23.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T23:52:23.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T23:52:23.765 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T23:52:23.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T23:52:23.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T23:52:23.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T23:52:23.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T23:52:23.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T23:52:23.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T23:52:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T23:52:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T23:52:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T23:52:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T23:52:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T23:52:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T23:52:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T23:52:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T23:52:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T23:52:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T23:52:23.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T23:52:23.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T23:52:23.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T23:52:23.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T23:52:23.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T23:52:23.771 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T23:52:23.942 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T23:52:23.942 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T23:52:23.942 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T23:52:23.942 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T23:52:23.942 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.table_properties_collectors: 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.inplace_update_support: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.bloom_locality: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.max_successive_merges: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T23:52:23.943 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.ttl: 2592000 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enable_blob_files: false 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.min_blob_size: 0 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T23:52:23.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6ebf5870-5158-4607-af8d-a216101df489 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178743692924, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T23:52:23.945 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T23:52:24.054 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178743693354, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700178743, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6ebf5870-5158-4607-af8d-a216101df489", "db_session_id": "C1A3P9HQQB3JB3PUJ40C", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T23:52:24.054 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700178743693430, "job": 1, "event": "recovery_finished"} 2023-11-16T23:52:24.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T23:52:24.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T23:52:24.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560648dc4000 2023-11-16T23:52:24.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: DB pointer 0x560648db0000 2023-11-16T23:52:24.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T23:52:24.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T23:52:24.212 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** DB Stats ** 2023-11-16T23:52:24.212 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:52:24.212 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:24.212 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:24.212 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:24.213 INFO:journalctl@ceph.mon.b.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T23:52:24.214 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache BinnedLRUCache@0x560648d271f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T23:52:24.215 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-11-16T23:52:24.258 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-11-16T23:52:24.258 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: starting mon.b rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:24.258 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(???) e0 preinit fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:24.258 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).mds e1 new map 2023-11-16T23:52:24.259 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).mds e1 print_map 2023-11-16T23:52:24.259 INFO:journalctl@ceph.mon.b.smithi156.stdout: e1 2023-11-16T23:52:24.259 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T23:52:24.259 INFO:journalctl@ceph.mon.b.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T23:52:24.259 INFO:journalctl@ceph.mon.b.smithi156.stdout: legacy client fscid: -1 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout: No filesystems configured 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T23:52:24.260 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T23:52:24.261 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T23:52:24.261 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T23:52:24.261 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T23:52:24.261 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:24.261 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:24.262 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T23:52:24.262 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:24.262 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:24.262 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:24.262 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:24.263 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:52:36.335 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.335 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.335 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.337 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.337 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Deploying cephadm binary to smithi173 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Added host smithi173 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.370 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.371 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4249602981' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:36.372 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c;count:3 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.373 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Deploying daemon mon.c on smithi173 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.a calling monitor election 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.c calling monitor election 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: fsmap 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:36.440 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mgrmap e13: x(active, since 49s) 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: overall HEALTH_OK 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: Deploying daemon mon.b on smithi156 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/496705776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expand map: {default=false} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta from 'false' to 'false' 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expanded map: {default=false} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expand map: {default=info} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta from 'info' to 'info' 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expanded map: {default=info} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expand map: {default=daemon} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expand map: {default=debug} 2023-11-16T23:52:36.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T23:52:36.443 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: expand_channel_meta expanded map: {default=debug} 2023-11-16T23:52:36.443 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 ceph-mon[131105]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-11-16T23:52:36.472 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:23 smithi156 systemd[1]: Started Ceph mon.b for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:52:36.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:23 smithi173 ceph-mon[131534]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:23 smithi023 ceph-mon[97824]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:36.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: mon.a calling monitor election 2023-11-16T23:52:36.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.474 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: mon.c calling monitor election 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: mon.b calling monitor election 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T23:52:36.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: fsmap 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: mgrmap e13: x(active, since 57s) 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: overall HEALTH_OK 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.476 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.508 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.508 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.508 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.508 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/1736690632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.510 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:52:30 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:52:30.714+0000 7f3b10c2d700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-16T23:52:36.511 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: mon.a calling monitor election 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: mon.c calling monitor election 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: mon.b calling monitor election 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.512 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: fsmap 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: mgrmap e13: x(active, since 57s) 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: overall HEALTH_OK 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.640 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.641 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.641 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.641 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/1736690632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:36.641 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:36.641 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:36.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi023:/etc/ceph/ceph.conf 2023-11-16T23:52:36.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:52:36.676 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:36.676 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:31 smithi156 ceph-mon[131105]: Updating smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.761 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:32 smithi156 ceph-mon[131105]: Reconfiguring daemon mon.a on smithi023 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.762 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:34 smithi156 ceph-mon[131105]: Reconfiguring mon.b (monmap changed)... 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:34 smithi156 ceph-mon[131105]: Reconfiguring daemon mon.b on smithi156 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: Reconfiguring mon.c (monmap changed)... 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: Reconfiguring daemon mon.c on smithi173 2023-11-16T23:52:36.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:35 smithi156 ceph-mon[131105]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.764 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi023:/etc/ceph/ceph.conf 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:52:36.765 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:31 smithi023 ceph-mon[97824]: Updating smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:32 smithi023 ceph-mon[97824]: Reconfiguring daemon mon.a on smithi023 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:36.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:36.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:34 smithi023 ceph-mon[97824]: Reconfiguring mon.b (monmap changed)... 2023-11-16T23:52:36.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:34 smithi023 ceph-mon[97824]: Reconfiguring daemon mon.b on smithi156 2023-11-16T23:52:36.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:36.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: Reconfiguring mon.c (monmap changed)... 2023-11-16T23:52:37.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:37.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:37.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: Reconfiguring daemon mon.c on smithi173 2023-11-16T23:52:37.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:35 smithi023 ceph-mon[97824]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.066 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:52:37.066 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 -- 172.21.15.173:0/1395167275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f086c0035c0 msgr2=0x7f086c005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:37.066 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 --2- 172.21.15.173:0/1395167275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f086c0035c0 0x7f086c005a40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0884073360 tx=0x7f0874014ec0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 -- 172.21.15.173:0/1395167275 shutdown_connections 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 --2- 172.21.15.173:0/1395167275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f086c0035c0 0x7f086c005a40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 -- 172.21.15.173:0/1395167275 >> 172.21.15.173:0/1395167275 conn(0x7f0884074c40 msgr2=0x7f0884075040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 -- 172.21.15.173:0/1395167275 shutdown_connections 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.054+0000 7f088a87e700 1 -- 172.21.15.173:0/1395167275 wait complete. 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 Processor -- start 2023-11-16T23:52:37.295 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 -- start start 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 0x7f088419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0884196010 con 0x7f088419a2f0 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.055+0000 7f088a87e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08841a2060 con 0x7f086c0035c0 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f088a87e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08841a21a0 con 0x7f0884195940 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51072/0 (socket says 172.21.15.173:51072) 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 -- 172.21.15.173:0/1736690632 learned_addr learned my addr 172.21.15.173:0/1736690632 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 0x7f088419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0883fff700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 msgr2=0x7f0884199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T23:52:37.296 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 msgr2=0x7f0884199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f08837fe700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0883fff700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 msgr2=0x7f0884195400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0883fff700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 msgr2=0x7f0884195400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0883fff700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0883fff700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 msgr2=0x7f0884199db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 msgr2=0x7f0884195400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 -- 172.21.15.173:0/1736690632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874014910 con 0x7f088419a2f0 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.056+0000 7f0888e1b700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 0x7f088419a790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f087c00be30 tx=0x7f087c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:37.297 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c00cc90 con 0x7f088419a2f0 2023-11-16T23:52:37.298 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f087c00cdf0 con 0x7f088419a2f0 2023-11-16T23:52:37.298 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0884107c80 con 0x7f088419a2f0 2023-11-16T23:52:37.298 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0127e0 con 0x7f088419a2f0 2023-11-16T23:52:37.298 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0884107af0 con 0x7f088419a2f0 2023-11-16T23:52:37.298 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.057+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f087c012960 con 0x7f088419a2f0 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.058+0000 7f08817fa700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 0x7f087003b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.058+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0884107e10 con 0x7f088419a2f0 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.058+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f087c04ac90 con 0x7f088419a2f0 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.058+0000 7f0883fff700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 0x7f087003b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.059+0000 7f0883fff700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 0x7f087003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0874010010 tx=0x7f0874001e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:37.388 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.063+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f087c019070 con 0x7f088419a2f0 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: mon.a calling monitor election 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: mon.c calling monitor election 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: mon.b calling monitor election 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: fsmap 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: mgrmap e13: x(active, since 57s) 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: overall HEALTH_OK 2023-11-16T23:52:37.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.391 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.391 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.339+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0884062120 con 0x7f088419a2f0 2023-11-16T23:52:37.391 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.339+0000 7f08817fa700 1 -- 172.21.15.173:0/1736690632 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1276 (secure 0 0 0) 0x7f087c01d3b0 con 0x7f088419a2f0 2023-11-16T23:52:37.391 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-16T23:52:37.391 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T23:52:23.715119Z","created":"2023-11-16T23:50:29.183152Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-11-16T23:52:37.393 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 msgr2=0x7f087003b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:37.393 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 0x7f087003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0874010010 tx=0x7f0874001e50 comp rx=0 tx=0).stop 2023-11-16T23:52:37.393 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 msgr2=0x7f088419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 0x7f088419a790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f087c00be30 tx=0x7f087c00bf10 comp rx=0 tx=0).stop 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 shutdown_connections 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088419a2f0 0x7f088419a790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f086c0035c0 0x7f0884195400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0870038d40 0x7f087003b1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.428 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 --2- 172.21.15.173:0/1736690632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0884195940 0x7f0884199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:37.429 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 >> 172.21.15.173:0/1736690632 conn(0x7f0884074c40 msgr2=0x7f088418d260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:37.429 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 shutdown_connections 2023-11-16T23:52:37.429 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:29.343+0000 7f088a87e700 1 -- 172.21.15.173:0/1736690632 wait complete. 2023-11-16T23:52:37.429 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2023-11-16T23:52:37.429 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-16T23:52:37.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-16T23:52:37.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/1736690632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi023:/etc/ceph/ceph.conf 2023-11-16T23:52:37.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi156:/etc/ceph/ceph.conf 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi173:/etc/ceph/ceph.conf 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:31 smithi173 ceph-mon[131534]: Updating smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.511 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:32 smithi173 ceph-mon[131534]: Reconfiguring daemon mon.a on smithi023 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:34 smithi173 ceph-mon[131534]: Reconfiguring mon.b (monmap changed)... 2023-11-16T23:52:37.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:34 smithi173 ceph-mon[131534]: Reconfiguring daemon mon.b on smithi156 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: Reconfiguring mon.c (monmap changed)... 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: Reconfiguring daemon mon.c on smithi173 2023-11-16T23:52:37.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:35 smithi173 ceph-mon[131534]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:37.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:37.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:37.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:37.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:37.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:52:38.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:38 smithi023 ceph-mon[97824]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:38 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:38 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:38.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:38 smithi173 ceph-mon[131534]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:38 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:38 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:38 smithi156 ceph-mon[131105]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:38 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:38 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:39 smithi023 ceph-mon[97824]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:39 smithi173 ceph-mon[131534]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:39.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:39 smithi156 ceph-mon[131105]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 -- 172.21.15.23:0/1469092227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 msgr2=0x7f685410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 --2- 172.21.15.23:0/1469092227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f685410d160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f684c00b0d0 tx=0x7f684c00b3e0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 -- 172.21.15.23:0/1469092227 shutdown_connections 2023-11-16T23:52:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 --2- 172.21.15.23:0/1469092227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f685410d160 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 --2- 172.21.15.23:0/1469092227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 0x7f6854100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 --2- 172.21.15.23:0/1469092227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 0x7f6854104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 -- 172.21.15.23:0/1469092227 >> 172.21.15.23:0/1469092227 conn(0x7f68540fbdc0 msgr2=0x7f68540fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 -- 172.21.15.23:0/1469092227 shutdown_connections 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.859+0000 7f685c709700 1 -- 172.21.15.23:0/1469092227 wait complete. 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.860+0000 7f685c709700 1 Processor -- start 2023-11-16T23:52:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 -- start start 2023-11-16T23:52:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 0x7f6854072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 0x7f6854073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6854073c20 con 0x7f68541055e0 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68541a4bd0 con 0x7f6854100ee0 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.861+0000 7f685c709700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68541a4d40 con 0x7f6854100540 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35370/0 (socket says 172.21.15.23:35370) 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 -- 172.21.15.23:0/4161993794 learned_addr learned my addr 172.21.15.23:0/4161993794 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685a4a5700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 0x7f6854072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f6859ca4700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 0x7f6854073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:39.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 -- 172.21.15.23:0/4161993794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 msgr2=0x7f6854072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 0x7f6854072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 -- 172.21.15.23:0/4161993794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 msgr2=0x7f6854073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 0x7f6854073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 -- 172.21.15.23:0/4161993794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c009d00 con 0x7f68541055e0 2023-11-16T23:52:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.862+0000 7f685aca6700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f684c0077f0 tx=0x7f684c007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.863+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c00e040 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.863+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f684c003e10 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.863+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c01b990 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.863+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6854110370 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.863+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68541101e0 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.864+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f684c0055c0 con 0x7f68541055e0 2023-11-16T23:52:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.864+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6854110500 con 0x7f68541055e0 2023-11-16T23:52:39.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.864+0000 7f684b7fe700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 0x7f684003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:39.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.864+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f684c04c8f0 con 0x7f68541055e0 2023-11-16T23:52:39.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.865+0000 7f685a4a5700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 0x7f684003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:39.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.865+0000 7f685a4a5700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 0x7f684003ae90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6850005950 tx=0x7f685000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:39.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:39.870+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f684c011d60 con 0x7f68541055e0 2023-11-16T23:52:40.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.078+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6854066ea0 con 0x7f68541055e0 2023-11-16T23:52:40.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.078+0000 7f684b7fe700 1 -- 172.21.15.23:0/4161993794 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f684c01f070 con 0x7f68541055e0 2023-11-16T23:52:40.080 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:40.080 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2023-11-16T23:52:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-16T23:52:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 msgr2=0x7f684003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 0x7f684003ae90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6850005950 tx=0x7f685000b410 comp rx=0 tx=0).stop 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 msgr2=0x7f68541a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f684c0077f0 tx=0x7f684c007820 comp rx=0 tx=0).stop 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 shutdown_connections 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68541055e0 0x7f68541a4690 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f68400389e0 0x7f684003ae90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6854100540 0x7f6854072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:40.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 --2- 172.21.15.23:0/4161993794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6854100ee0 0x7f6854073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 >> 172.21.15.23:0/4161993794 conn(0x7f68540fbdc0 msgr2=0x7f68540fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 shutdown_connections 2023-11-16T23:52:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:40.081+0000 7f685c709700 1 -- 172.21.15.23:0/4161993794 wait complete. 2023-11-16T23:52:40.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4161993794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:40.555 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-16T23:52:40.555 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:52:40.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T23:52:40.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:40 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4161993794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:40.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:40 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4161993794' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:40.633 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:52:40.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:40.743 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:52:40.743 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T23:52:40.779 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:52:40.779 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:40.851 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:52:40.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T23:52:40.888 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:52:40.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T23:52:40.960 INFO:tasks.cephadm:Adding mgr.x on smithi023 2023-11-16T23:52:40.960 INFO:tasks.cephadm:Adding mgr.y on smithi156 2023-11-16T23:52:40.960 INFO:tasks.cephadm:Adding mgr.z on smithi173 2023-11-16T23:52:40.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch apply mgr '3;smithi023=x;smithi156=y;smithi173=z' 2023-11-16T23:52:41.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:41 smithi173 ceph-mon[131534]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:41.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:41 smithi156 ceph-mon[131105]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:41.636 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:52:41.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:41 smithi023 ceph-mon[97824]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.857+0000 7f1268f4f700 1 -- 172.21.15.173:0/3586943551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12641002f0 msgr2=0x7f1264100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.857+0000 7f1268f4f700 1 --2- 172.21.15.173:0/3586943551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12641002f0 0x7f1264100700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1258009990 tx=0x7f1258009ca0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 -- 172.21.15.173:0/3586943551 shutdown_connections 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 --2- 172.21.15.173:0/3586943551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1264101e80 0x7f1264108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 --2- 172.21.15.173:0/3586943551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f12641002f0 0x7f1264100700 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 --2- 172.21.15.173:0/3586943551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641014f0 0x7f1264101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 -- 172.21.15.173:0/3586943551 >> 172.21.15.173:0/3586943551 conn(0x7f12640fb900 msgr2=0x7f12640fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 -- 172.21.15.173:0/3586943551 shutdown_connections 2023-11-16T23:52:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 -- 172.21.15.173:0/3586943551 wait complete. 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 Processor -- start 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 -- start start 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641002f0 0x7f1264196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.858+0000 7f1268f4f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12641014f0 0x7f1264196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:42.863 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126419dbd0 con 0x7f12641002f0 2023-11-16T23:52:42.901 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126419dd10 con 0x7f12641014f0 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126419dea0 con 0x7f1264101e80 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51086/0 (socket says 172.21.15.173:51086) 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 -- 172.21.15.173:0/985837693 learned_addr learned my addr 172.21.15.173:0/985837693 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1261d9b700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12641014f0 0x7f1264196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f126259c700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641002f0 0x7f1264196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 -- 172.21.15.173:0/985837693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12641014f0 msgr2=0x7f1264196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12641014f0 0x7f1264196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 -- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641002f0 msgr2=0x7f1264196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641002f0 0x7f1264196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:42.902 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 -- 172.21.15.173:0/985837693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580095f0 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1262d9d700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f125c00add0 tx=0x7f125c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c00f070 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126410bcb0 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f125c007b30 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c005310 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.859+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126410bb20 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.860+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f125c00c940 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.860+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12641078f0 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.861+0000 7f12537fe700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 0x7f124c03ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.861+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f125c04a930 con 0x7f1264101e80 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.861+0000 7f126259c700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 0x7f124c03ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:42.903 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.862+0000 7f126259c700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 0x7f124c03ae90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f125800b5c0 tx=0x7f1258005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:42.904 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:42.865+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f125c01a310 con 0x7f1264101e80 2023-11-16T23:52:43.060 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.059+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi156=y;smithi173=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f1264061c70 con 0x7f124c0389e0 2023-11-16T23:52:43.067 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.067+0000 7f12537fe700 1 -- 172.21.15.173:0/985837693 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1264061c70 con 0x7f124c0389e0 2023-11-16T23:52:43.068 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2023-11-16T23:52:43.070 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 msgr2=0x7f124c03ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:43.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 0x7f124c03ae90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f125800b5c0 tx=0x7f1258005fb0 comp rx=0 tx=0).stop 2023-11-16T23:52:43.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 msgr2=0x7f126419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:43.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f125c00add0 tx=0x7f125c00c330 comp rx=0 tx=0).stop 2023-11-16T23:52:43.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 shutdown_connections 2023-11-16T23:52:43.071 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f124c0389e0 0x7f124c03ae90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1264101e80 0x7f126419d600 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12641002f0 0x7f1264196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 --2- 172.21.15.173:0/985837693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12641014f0 0x7f1264196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 >> 172.21.15.173:0/985837693 conn(0x7f12640fb900 msgr2=0x7f1264104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 shutdown_connections 2023-11-16T23:52:43.072 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:52:43.069+0000 7f1268f4f700 1 -- 172.21.15.173:0/985837693 wait complete. 2023-11-16T23:52:43.597 DEBUG:teuthology.orchestra.run.smithi156:mgr.y> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.y.service 2023-11-16T23:52:43.600 DEBUG:teuthology.orchestra.run.smithi173:mgr.z> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.z.service 2023-11-16T23:52:43.602 INFO:tasks.cephadm:Deploying OSDs... 2023-11-16T23:52:43.602 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:52:43.602 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T23:52:43.631 INFO:journalctl@ceph.mgr.y.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:52:43.634 INFO:journalctl@ceph.mgr.z.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:52:43.650 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T23:52:43.651 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:43.739 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-11-16 23:51:40.826274416 +0000 2023-11-16T23:52:43.740 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-11-16 23:45:53.992503431 +0000 2023-11-16T23:52:43.740 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-11-16 23:45:53.992503431 +0000 2023-11-16T23:52:43.740 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-11-16 23:45:53.961503673 +0000 2023-11-16T23:52:43.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T23:52:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-11-16T23:52:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-11-16T23:52:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000438933 s, 1.2 MB/s 2023-11-16T23:52:43.852 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T23:52:43.898 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2023-11-16T23:52:43.942 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T23:52:43.942 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:43.942 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-11-16T23:52:43.942 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:43.943 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:43.943 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-11-16 23:51:40.826274416 +0000 2023-11-16T23:52:43.943 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-11-16 23:45:54.864496613 +0000 2023-11-16T23:52:43.943 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-11-16 23:45:54.864496613 +0000 2023-11-16T23:52:43.943 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-11-16 23:45:54.832496863 +0000 2023-11-16T23:52:43.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T23:52:44.049 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-11-16T23:52:44.049 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-11-16T23:52:44.049 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000465792 s, 1.1 MB/s 2023-11-16T23:52:44.052 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T23:52:44.098 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2023-11-16T23:52:44.143 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T23:52:44.143 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:44.143 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-11-16 23:51:40.826274416 +0000 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-11-16 23:45:55.712489983 +0000 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-11-16 23:45:55.712489983 +0000 2023-11-16T23:52:44.144 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-11-16 23:45:55.684490202 +0000 2023-11-16T23:52:44.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi156=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: Saving service mgr spec with placement smithi023=x;smithi156=y;smithi173=z;count:3 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.219 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.220 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:44 smithi173 ceph-mon[131534]: Deploying daemon mgr.z on smithi173 2023-11-16T23:52:44.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi156=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: Saving service mgr spec with placement smithi023=x;smithi156=y;smithi173=z;count:3 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:44.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:44.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:44 smithi023 ceph-mon[97824]: Deploying daemon mgr.z on smithi173 2023-11-16T23:52:44.249 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-11-16T23:52:44.250 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-11-16T23:52:44.250 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000359598 s, 1.4 MB/s 2023-11-16T23:52:44.253 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T23:52:44.298 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi156=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: Saving service mgr spec with placement smithi023=x;smithi156=y;smithi173=z;count:3 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:44.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:44 smithi156 ceph-mon[131105]: Deploying daemon mgr.z on smithi173 2023-11-16T23:52:44.382 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T23:52:44.382 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 470 Links: 1 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-11-16 23:51:40.826274416 +0000 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-11-16 23:45:56.558483368 +0000 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-11-16 23:45:56.558483368 +0000 2023-11-16T23:52:44.383 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-11-16 23:45:56.528483603 +0000 2023-11-16T23:52:44.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T23:52:44.489 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-11-16T23:52:44.489 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-11-16T23:52:44.489 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000382148 s, 1.3 MB/s 2023-11-16T23:52:44.491 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T23:52:44.538 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-16T23:52:44.538 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T23:52:44.562 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T23:52:44.563 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-11-16T23:52:44.623 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T23:52:44.623 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 389419 Links: 1 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-16 23:52:23.091136334 +0000 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-16 23:42:33.451219632 +0000 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-16 23:42:33.451219632 +0000 2023-11-16T23:52:44.624 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-16T23:52:44.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T23:52:44.693 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-16T23:52:44.693 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-16T23:52:44.694 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.00012937 s, 4.0 MB/s 2023-11-16T23:52:44.694 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T23:52:44.755 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 393456 Links: 1 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-16 23:52:23.091136334 +0000 2023-11-16T23:52:44.814 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-16 23:42:33.979221234 +0000 2023-11-16T23:52:44.815 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-16 23:42:33.979221234 +0000 2023-11-16T23:52:44.815 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-16T23:52:44.815 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T23:52:44.857 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:44 smithi173 systemd[1]: Starting Ceph mgr.z for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:52:44.882 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-16T23:52:44.882 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-16T23:52:44.883 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000127512 s, 4.0 MB/s 2023-11-16T23:52:44.884 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T23:52:44.944 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-11-16T23:52:45.053 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T23:52:45.053 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:45.053 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 392634 Links: 1 2023-11-16T23:52:45.053 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:45.053 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:45.054 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-16 23:52:23.091136334 +0000 2023-11-16T23:52:45.054 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-16 23:42:34.515222861 +0000 2023-11-16T23:52:45.054 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-16 23:42:34.515222861 +0000 2023-11-16T23:52:45.054 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-16T23:52:45.055 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T23:52:45.120 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-16T23:52:45.182 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-16T23:52:45.183 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000121835 s, 4.2 MB/s 2023-11-16T23:52:45.184 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T23:52:45.244 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-11-16T23:52:45.304 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T23:52:45.304 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:45.304 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 392695 Links: 1 2023-11-16T23:52:45.304 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:45.305 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:45.305 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-16 23:52:23.091136334 +0000 2023-11-16T23:52:45.305 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-16 23:42:35.049224482 +0000 2023-11-16T23:52:45.305 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-16 23:42:35.049224482 +0000 2023-11-16T23:52:45.305 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-11-16T23:52:45.306 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T23:52:45.333 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 podman[133414]: 2023-11-16T23:52:45.373 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-16T23:52:45.393 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-16T23:52:45.393 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000115956 s, 4.4 MB/s 2023-11-16T23:52:45.396 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T23:52:45.455 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-16T23:52:45.455 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T23:52:45.473 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T23:52:45.474 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2023-11-16T23:52:45.531 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T23:52:45.769 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:45.769 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 384688 Links: 1 2023-11-16T23:52:45.770 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:45.770 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:45.770 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-16 23:52:44.678131743 +0000 2023-11-16T23:52:45.770 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-16 23:42:36.455302060 +0000 2023-11-16T23:52:45.770 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-16 23:42:36.455302060 +0000 2023-11-16T23:52:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-16T23:52:45.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T23:52:45.859 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 bash[133414]: 884f6816a04e7a7365768a5238043122772c6eedeced17c2afdf158920b46985 2023-11-16T23:52:45.859 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-mgr[133454]: -- 172.21.15.173:0/274643985 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5618d70af520 con 0x5618d7085000 2023-11-16T23:52:45.859 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 systemd[1]: Started Ceph mgr.z for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:52:45.860 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:45.458+0000 7f6ea2560200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T23:52:45.860 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:45.547+0000 7f6ea2560200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T23:52:45.886 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:45.635+0000 7f6ea2560200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T23:52:45.886 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:45.726+0000 7f6ea2560200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T23:52:45.887 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-16T23:52:45.887 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-16T23:52:45.887 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000245404 s, 2.1 MB/s 2023-11-16T23:52:45.888 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T23:52:45.948 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 390931 Links: 1 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-16 23:52:44.678131743 +0000 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-16 23:42:37.059288218 +0000 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-16 23:42:37.059288218 +0000 2023-11-16T23:52:46.007 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-16T23:52:46.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T23:52:46.074 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-16T23:52:46.074 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-16T23:52:46.074 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000215782 s, 2.4 MB/s 2023-11-16T23:52:46.075 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T23:52:46.132 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 393367 Links: 1 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-16 23:52:44.678131743 +0000 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-16 23:42:37.608275636 +0000 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-16 23:42:37.608275636 +0000 2023-11-16T23:52:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-16T23:52:46.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T23:52:46.226 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:45.895+0000 7f6ea2560200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:46.227 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:46.251 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-16T23:52:46.252 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-16T23:52:46.252 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000202541 s, 2.5 MB/s 2023-11-16T23:52:46.253 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T23:52:46.311 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2023-11-16T23:52:46.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:46.370 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T23:52:46.370 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T23:52:46.370 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 393527 Links: 1 2023-11-16T23:52:46.370 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T23:52:46.370 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T23:52:46.371 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-16 23:52:44.678131743 +0000 2023-11-16T23:52:46.371 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-16 23:42:38.162262940 +0000 2023-11-16T23:52:46.371 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-16 23:42:38.162262940 +0000 2023-11-16T23:52:46.371 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-16T23:52:46.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T23:52:46.437 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-16T23:52:46.438 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-16T23:52:46.438 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000235205 s, 2.2 MB/s 2023-11-16T23:52:46.439 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T23:52:46.498 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2023-11-16T23:52:46.499 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T23:52:46.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:46.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:46.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:46.225+0000 7f6ea2560200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T23:52:46.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:46.319+0000 7f6ea2560200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T23:52:46.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:46.456+0000 7f6ea2560200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T23:52:46.856 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:46.556+0000 7f6ea2560200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T23:52:47.081 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:46 smithi156 systemd[1]: Starting Ceph mgr.y for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:52:47.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:47 smithi173 ceph-mon[131534]: Deploying daemon mgr.y on smithi156 2023-11-16T23:52:47.356 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:46.974+0000 7f6ea2560200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T23:52:47.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-mon[131105]: Deploying daemon mgr.y on smithi156 2023-11-16T23:52:47.357 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 podman[132578]: 2023-11-16T23:52:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:52:47.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:47 smithi023 ceph-mon[97824]: Deploying daemon mgr.y on smithi156 2023-11-16T23:52:47.815 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 bash[132578]: 331a0e3ea21d7e36d109c7567c1340e6379c6ed3d8e01a3078bcdf868cdd4149 2023-11-16T23:52:47.815 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-mgr[132618]: -- 172.21.15.156:0/3685110032 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5624eee19520 con 0x5624eedef000 2023-11-16T23:52:47.815 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 systemd[1]: Started Ceph mgr.y for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:52:47.880 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:47.645+0000 7fce29386200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T23:52:47.880 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:47.724+0000 7fce29386200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T23:52:47.880 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:47.814+0000 7fce29386200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T23:52:47.881 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:47.550+0000 7f6ea2560200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T23:52:47.881 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:47.724+0000 7f6ea2560200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T23:52:48.086 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:47.913+0000 7fce29386200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T23:52:48.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:48.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:48.357 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:48.099+0000 7fce29386200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:48.752 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:48.423+0000 7fce29386200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T23:52:48.752 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:48.517+0000 7fce29386200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T23:52:48.752 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:48.656+0000 7fce29386200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T23:52:48.753 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:48.751+0000 7fce29386200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T23:52:48.818 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:48.548+0000 7f6ea2560200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:52:48.818 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:48.640+0000 7f6ea2560200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T23:52:49.087 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:48.817+0000 7f6ea2560200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:52:49.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:49 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:49.345 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:49.108+0000 7f6ea2560200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T23:52:49.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:49 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:49.357 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:49.202+0000 7fce29386200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T23:52:49.433 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T23:52:49.433 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T23:52:49.433 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2023-11-16T23:52:49.434 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2023-11-16T23:52:49.434 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0507662 s, 207 MB/s 2023-11-16T23:52:49.434 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2023-11-16T23:52:49.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:49 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:49.606 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:49.344+0000 7f6ea2560200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T23:52:49.606 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:49.497+0000 7f6ea2560200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T23:52:49.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2023-11-16T23:52:50.091 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:49.812+0000 7fce29386200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T23:52:50.091 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:49.982+0000 7fce29386200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T23:52:50.091 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:49.760+0000 7f6ea2560200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T23:52:50.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:50.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T23:52:50.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:50.780 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:50.460+0000 7f6ea2560200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T23:52:50.780 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:50.589+0000 7f6ea2560200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T23:52:50.942 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:52:51.003 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:50.744+0000 7fce29386200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:52:51.003 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:50.833+0000 7fce29386200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T23:52:51.095 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:50.778+0000 7f6ea2560200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T23:52:51.095 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:50.890+0000 7f6ea2560200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:51.002+0000 7fce29386200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: Reconfiguring daemon mgr.x on smithi023 2023-11-16T23:52:51.278 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:51.353 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:52:51.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T23:52:51.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: Reconfiguring daemon mgr.x on smithi023 2023-11-16T23:52:51.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:51 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:51.356 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:51.105+0000 7f6ea2560200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T23:52:51.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:52:51.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:51.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T23:52:51.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: Reconfiguring daemon mgr.x on smithi023 2023-11-16T23:52:51.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:51.606 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:51.276+0000 7fce29386200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T23:52:51.607 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:51.504+0000 7fce29386200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T23:52:51.912 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:51.650+0000 7fce29386200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T23:52:52.356 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 16 23:52:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z[133430]: 2023-11-16T23:52:51.958+0000 7f6ea2560200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: Standby manager daemon z started 2023-11-16T23:52:52.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T23:52:52.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:52.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T23:52:52.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:52 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:51.911+0000 7fce29386200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:52.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: Standby manager daemon z started 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T23:52:52.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:52.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: Standby manager daemon z started 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T23:52:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:52 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.173:0/1066189252' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 -- 172.21.15.23:0/3932423464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1524101e80 msgr2=0x7f1524108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 --2- 172.21.15.23:0/3932423464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1524101e80 0x7f1524108740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f151c009f90 tx=0x7f151c00b330 comp rx=0 tx=0).stop 2023-11-16T23:52:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 -- 172.21.15.23:0/3932423464 shutdown_connections 2023-11-16T23:52:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 --2- 172.21.15.23:0/3932423464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1524101e80 0x7f1524108740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 --2- 172.21.15.23:0/3932423464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15241002f0 0x7f1524100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 --2- 172.21.15.23:0/3932423464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 0x7f1524101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 -- 172.21.15.23:0/3932423464 >> 172.21.15.23:0/3932423464 conn(0x7f15240fb900 msgr2=0x7f15240fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:52:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 -- 172.21.15.23:0/3932423464 shutdown_connections 2023-11-16T23:52:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.796+0000 7f1529f25700 1 -- 172.21.15.23:0/3932423464 wait complete. 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.797+0000 7f1529f25700 1 Processor -- start 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.797+0000 7f1529f25700 1 -- start start 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 0x7f1524196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1524101e80 0x7f152419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152419dce0 con 0x7f15241002f0 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152419de20 con 0x7f1524101e80 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1529f25700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152419dfb0 con 0x7f15241014f0 2023-11-16T23:52:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f15237fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f15237fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51166/0 (socket says 172.21.15.23:51166) 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f15237fe700 1 -- 172.21.15.23:0/2121574234 learned_addr learned my addr 172.21.15.23:0/2121574234 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1523fff700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1524101e80 0x7f152419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.798+0000 7f1522ffd700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 0x7f1524196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 -- 172.21.15.23:0/2121574234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 msgr2=0x7f1524196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 0x7f1524196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 -- 172.21.15.23:0/2121574234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1524101e80 msgr2=0x7f152419d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1524101e80 0x7f152419d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:52:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 -- 172.21.15.23:0/2121574234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15140095f0 con 0x7f15241002f0 2023-11-16T23:52:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f15237fe700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f151400ec20 tx=0x7f151400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514009a10 con 0x7f15241002f0 2023-11-16T23:52:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f151400cc90 con 0x7f15241002f0 2023-11-16T23:52:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15140187e0 con 0x7f15241002f0 2023-11-16T23:52:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c009c40 con 0x7f15241002f0 2023-11-16T23:52:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.799+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152410bcb0 con 0x7f15241002f0 2023-11-16T23:52:52.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.800+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1524107860 con 0x7f15241002f0 2023-11-16T23:52:52.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.801+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f15140109f0 con 0x7f15241002f0 2023-11-16T23:52:52.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.802+0000 7f1520ff9700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 0x7f150c06f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:52:52.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.802+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1514014070 con 0x7f15241002f0 2023-11-16T23:52:52.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.802+0000 7f1522ffd700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 0x7f150c06f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:52:52.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.803+0000 7f1522ffd700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 0x7f150c06f200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1518009c60 tx=0x7f1518009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:52:52.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:52.807+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1514056b00 con 0x7f15241002f0 2023-11-16T23:52:52.939 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:52.617+0000 7fce29386200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T23:52:52.939 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:52.749+0000 7fce29386200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T23:52:53.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:53.025+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1524060f30 con 0x7f150c06cd50 2023-11-16T23:52:53.267 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:52.938+0000 7fce29386200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T23:52:53.267 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:53.051+0000 7fce29386200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: mgrmap e14: x(active, since 81s), standbys: z 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:52:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:53.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: mgrmap e14: x(active, since 81s), standbys: z 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:52:53.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:52:53.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: mgrmap e14: x(active, since 81s), standbys: z 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:52:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:52:53.607 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:53.266+0000 7fce29386200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T23:52:54.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:54 smithi173 ceph-mon[131534]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:54.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:54 smithi173 ceph-mon[131534]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:54.357 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 16 23:52:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y[132594]: 2023-11-16T23:52:54.117+0000 7fce29386200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T23:52:54.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:54 smithi156 ceph-mon[131105]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:54.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:54 smithi156 ceph-mon[131105]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:54.385 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:54 smithi023 ceph-mon[97824]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:54.385 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:54 smithi023 ceph-mon[97824]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:52:55.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:52:55.116+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f1514056ed0 con 0x7f15241002f0 2023-11-16T23:52:55.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:55 smithi023 ceph-mon[97824]: Standby manager daemon y started 2023-11-16T23:52:55.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:55 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T23:52:55.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:55 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:55.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:55 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T23:52:55.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:55 smithi023 ceph-mon[97824]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:55.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:55 smithi173 ceph-mon[131534]: Standby manager daemon y started 2023-11-16T23:52:55.618 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:55 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:55 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:55 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:55 smithi173 ceph-mon[131534]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:55 smithi156 ceph-mon[131105]: Standby manager daemon y started 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:55 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:55 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T23:52:55.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:55 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T23:52:55.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:55 smithi156 ceph-mon[131105]: from='mgr.? 172.21.15.156:0/3740595820' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T23:52:56.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:56 smithi023 ceph-mon[97824]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:56.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:56 smithi023 ceph-mon[97824]: mgrmap e15: x(active, since 84s), standbys: z, y 2023-11-16T23:52:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T23:52:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:56 smithi173 ceph-mon[131534]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:56 smithi173 ceph-mon[131534]: mgrmap e15: x(active, since 84s), standbys: z, y 2023-11-16T23:52:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T23:52:56.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:56 smithi156 ceph-mon[131105]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:56.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:56 smithi156 ceph-mon[131105]: mgrmap e15: x(active, since 84s), standbys: z, y 2023-11-16T23:52:56.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T23:52:57.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:57 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]: dispatch 2023-11-16T23:52:57.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:57 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]': finished 2023-11-16T23:52:57.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:57 smithi023 ceph-mon[97824]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T23:52:57.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:52:57.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:57 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2650120372' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:52:57.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:57 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]: dispatch 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:57 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]': finished 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:57 smithi173 ceph-mon[131534]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:57 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2650120372' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:57 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]: dispatch 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:57 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2993475376' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823ad49-73e7-442f-89dc-91875e102c7b"}]': finished 2023-11-16T23:52:57.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:57 smithi156 ceph-mon[131105]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T23:52:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:52:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:57 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2650120372' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:52:58.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:58 smithi023 ceph-mon[97824]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:58 smithi173 ceph-mon[131534]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:58 smithi156 ceph-mon[131105]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:59.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:52:59 smithi173 ceph-mon[131534]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:59.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:52:59 smithi156 ceph-mon[131105]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:52:59.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:52:59 smithi023 ceph-mon[97824]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:02.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:02 smithi023 ceph-mon[97824]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:02.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:02 smithi156 ceph-mon[131105]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:02 smithi173 ceph-mon[131534]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:04.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:04 smithi023 ceph-mon[97824]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:04.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T23:53:04.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:04.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:04 smithi173 ceph-mon[131534]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:04.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T23:53:04.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:04.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:04 smithi156 ceph-mon[131105]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:04.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T23:53:04.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:05.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:05 smithi023 ceph-mon[97824]: Deploying daemon osd.0 on smithi023 2023-11-16T23:53:05.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:05 smithi173 ceph-mon[131534]: Deploying daemon osd.0 on smithi023 2023-11-16T23:53:05.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:05 smithi156 ceph-mon[131105]: Deploying daemon osd.0 on smithi023 2023-11-16T23:53:06.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:06 smithi173 ceph-mon[131534]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:06 smithi156 ceph-mon[131105]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:06.491 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:06 smithi023 ceph-mon[97824]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:08.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:08 smithi173 ceph-mon[131534]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:08.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:08 smithi156 ceph-mon[131105]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:08.365 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:08 smithi023 ceph-mon[97824]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:09.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:09 smithi023 ceph-mon[97824]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:09.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:09 smithi173 ceph-mon[131534]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:09.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:09 smithi156 ceph-mon[131105]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:11.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:11.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:11 smithi023 ceph-mon[97824]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:11.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:11 smithi173 ceph-mon[131534]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:11.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:11 smithi156 ceph-mon[131105]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:14.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.184+0000 7f1520ff9700 1 -- 172.21.15.23:0/2121574234 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1524060f30 con 0x7f150c06cd50 2023-11-16T23:53:14.185 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 msgr2=0x7f150c06f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 0x7f150c06f200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1518009c60 tx=0x7f1518009260 comp rx=0 tx=0).stop 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 msgr2=0x7f1524196150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f151400ec20 tx=0x7f151400ef30 comp rx=0 tx=0).stop 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 shutdown_connections 2023-11-16T23:53:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1524101e80 0x7f152419d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15241002f0 0x7f1524196150 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f150c06cd50 0x7f150c06f200 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 --2- 172.21.15.23:0/2121574234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15241014f0 0x7f1524196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.186+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 >> 172.21.15.23:0/2121574234 conn(0x7f15240fb900 msgr2=0x7f15241049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.187+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 shutdown_connections 2023-11-16T23:53:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:53:14.187+0000 7f1529f25700 1 -- 172.21.15.23:0/2121574234 wait complete. 2023-11-16T23:53:14.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:14.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:53:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:14 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:14 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:14.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:53:14.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:14 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:14.636 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.0.service 2023-11-16T23:53:14.640 INFO:tasks.cephadm:Deploying osd.1 on smithi156 with /dev/vg_nvme/lv_4... 2023-11-16T23:53:14.641 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T23:53:14.718 INFO:journalctl@ceph.osd.0.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:53:15.288 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:53:15.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:15.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:15 smithi023 ceph-mon[97824]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T23:53:15.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:15.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:15 smithi156 ceph-mon[131105]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T23:53:15.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:15.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:15.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:15 smithi173 ceph-mon[131534]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T23:53:16.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-mon[97824]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:16.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-mon[97824]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T23:53:16.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-mon[97824]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T23:53:16.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:16.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-mon[97824]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:53:16.319 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0[110064]: 2023-11-16T23:53:16.178+0000 7f837285e700 -1 osd.0 0 waiting for initial osdmap 2023-11-16T23:53:16.319 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:53:16 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0[110064]: 2023-11-16T23:53:16.184+0000 7f836de52700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:53:16.437 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:16 smithi156 ceph-mon[131105]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:16.437 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:16 smithi156 ceph-mon[131105]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T23:53:16.437 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:16 smithi156 ceph-mon[131105]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T23:53:16.437 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:16.437 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:16 smithi156 ceph-mon[131105]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:53:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:16 smithi173 ceph-mon[131534]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:16 smithi173 ceph-mon[131534]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T23:53:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:16 smithi173 ceph-mon[131534]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T23:53:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:16 smithi173 ceph-mon[131534]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:53:16.921 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T23:53:16.921 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T23:53:16.921 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-11-16T23:53:16.921 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-11-16T23:53:16.921 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00810023 s, 1.3 GB/s 2023-11-16T23:53:16.922 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-11-16T23:53:17.329 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:17 smithi156 ceph-mon[131105]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:53:17.329 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:17 smithi156 ceph-mon[131105]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T23:53:17.329 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:17 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.329 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:17 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.347 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2023-11-16T23:53:17.433 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:17 smithi023 ceph-mon[97824]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:53:17.434 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:17 smithi023 ceph-mon[97824]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T23:53:17.434 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:17 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.434 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:17 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:17 smithi173 ceph-mon[131534]: from='osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:53:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:17 smithi173 ceph-mon[131534]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T23:53:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:17 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:17 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:17.987 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] boot 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:18.471 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.472 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:18.472 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:18.472 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:53:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:53:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] boot 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:18.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: osd.0 [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] boot 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:18.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:18.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:18.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:19.180 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- 172.21.15.156:0/2538329613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a00ff420 msgr2=0x7f31a00ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:19.180 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 --2- 172.21.15.156:0/2538329613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a00ff420 0x7f31a00ff830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194009ca0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.180 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- 172.21.15.156:0/2538329613 shutdown_connections 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 --2- 172.21.15.156:0/2538329613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 0x7f31a0107870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 --2- 172.21.15.156:0/2538329613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a00ff420 0x7f31a00ff830 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 --2- 172.21.15.156:0/2538329613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0100620 0x7f31a0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- 172.21.15.156:0/2538329613 >> 172.21.15.156:0/2538329613 conn(0x7f31a00faa50 msgr2=0x7f31a00fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- 172.21.15.156:0/2538329613 shutdown_connections 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- 172.21.15.156:0/2538329613 wait complete. 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 Processor -- start 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.177+0000 7f31a71c3700 1 -- start start 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a00ff420 0x7f31a01962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:19.181 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 0x7f31a019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019de40 con 0x7f31a0100fb0 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019df80 con 0x7f31a0100620 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019e110 con 0x7f31a00ff420 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35192/0 (socket says 172.21.15.156:35192) 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 -- 172.21.15.156:0/3771841277 learned_addr learned my addr 172.21.15.156:0/3771841277 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-16T23:53:19.182 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a4f5f700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a00ff420 0x7f31a01962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a5760700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 0x7f31a019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 -- 172.21.15.156:0/3771841277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a00ff420 msgr2=0x7f31a01962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a00ff420 0x7f31a01962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 -- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 msgr2=0x7f31a019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 0x7f31a019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 -- 172.21.15.156:0/3771841277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940095f0 con 0x7f31a0100620 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319ffff700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f318c00d950 tx=0x7f318c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c009940 con 0x7f31a0100620 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f318c010460 con 0x7f31a0100620 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a010adf0 con 0x7f31a0100620 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.178+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c00f600 con 0x7f31a0100620 2023-11-16T23:53:19.183 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.179+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a010ac60 con 0x7f31a0100620 2023-11-16T23:53:19.184 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.179+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a0067b00 con 0x7f31a0100620 2023-11-16T23:53:19.184 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.180+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f318c0105d0 con 0x7f31a0100620 2023-11-16T23:53:19.184 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.181+0000 7f319dffb700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 0x7f31900a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:19.185 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.181+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f318c0cada0 con 0x7f31a0100620 2023-11-16T23:53:19.185 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.181+0000 7f31a4f5f700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 0x7f31900a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:19.185 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.182+0000 7f31a4f5f700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 0x7f31900a35f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f319400b5c0 tx=0x7f3194005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:53:19.186 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.185+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f318c095ec0 con 0x7f31a0100620 2023-11-16T23:53:19.379 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:19.378+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a00697b0 con 0x7f31900a1140 2023-11-16T23:53:19.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi023 2023-11-16T23:53:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi023 to 17271M 2023-11-16T23:53:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:53:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:19 smithi023 ceph-mon[97824]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T23:53:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi023 2023-11-16T23:53:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi023 to 17271M 2023-11-16T23:53:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:53:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:19 smithi173 ceph-mon[131534]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T23:53:19.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi023 2023-11-16T23:53:19.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi023 to 17271M 2023-11-16T23:53:19.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:53:19.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:19.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:19 smithi156 ceph-mon[131105]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T23:53:20.483 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:20 smithi156 ceph-mon[131105]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:20.483 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:20 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:20.483 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:20 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:20.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:20 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:20.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:20 smithi023 ceph-mon[97824]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:20 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:20 smithi173 ceph-mon[131534]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:20.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:20.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:20 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:21.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:21 smithi023 ceph-mon[97824]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:21.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:21 smithi173 ceph-mon[131534]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:21.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:21 smithi156 ceph-mon[131105]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/2369920113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]': finished 2023-11-16T23:53:22.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T23:53:22.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/2369920113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]': finished 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/2369920113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]: dispatch 2023-11-16T23:53:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bd38f3-3da3-45e1-9d3b-d375d7fa95ef"}]': finished 2023-11-16T23:53:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T23:53:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:23.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:23 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/2266950287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:23.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:23 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/2266950287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:23.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/2266950287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:24.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:24 smithi023 ceph-mon[97824]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:24.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:24 smithi173 ceph-mon[131534]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:24 smithi156 ceph-mon[131105]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:26.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:26 smithi156 ceph-mon[131105]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:26.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:26 smithi023 ceph-mon[97824]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:26.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:26 smithi173 ceph-mon[131534]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:28 smithi023 ceph-mon[97824]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:28 smithi173 ceph-mon[131534]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:28 smithi156 ceph-mon[131105]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:29.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T23:53:29.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:29 smithi173 ceph-mon[131534]: Deploying daemon osd.1 on smithi156 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:29 smithi173 ceph-mon[131534]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:29 smithi156 ceph-mon[131105]: Deploying daemon osd.1 on smithi156 2023-11-16T23:53:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:29 smithi156 ceph-mon[131105]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:29.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T23:53:29.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:29.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:29 smithi023 ceph-mon[97824]: Deploying daemon osd.1 on smithi156 2023-11-16T23:53:29.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:29 smithi023 ceph-mon[97824]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:32.314 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:32 smithi156 ceph-mon[131105]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:32.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:32 smithi173 ceph-mon[131534]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:32.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:32 smithi023 ceph-mon[97824]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:34.493 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:34 smithi156 ceph-mon[131105]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:34.493 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.493 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.493 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:34 smithi023 ceph-mon[97824]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:34.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:34 smithi173 ceph-mon[131534]: pgmap v61: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:36.148 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.147+0000 7f319dffb700 1 -- 172.21.15.156:0/3771841277 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f31a00697b0 con 0x7f31900a1140 2023-11-16T23:53:36.148 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 1 on host 'smithi156' 2023-11-16T23:53:36.154 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 msgr2=0x7f31900a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:36.154 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 0x7f31900a35f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f319400b5c0 tx=0x7f3194005fd0 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 msgr2=0x7f31a01967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f318c00d950 tx=0x7f318c00dc60 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 shutdown_connections 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a0100fb0 0x7f31a019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f31900a1140 0x7f31900a35f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a00ff420 0x7f31a01962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 --2- 172.21.15.156:0/3771841277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31a0100620 0x7f31a01967f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 >> 172.21.15.156:0/3771841277 conn(0x7f31a00faa50 msgr2=0x7f31a00690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 shutdown_connections 2023-11-16T23:53:36.155 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:53:36.149+0000 7f31a71c3700 1 -- 172.21.15.156:0/3771841277 wait complete. 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: pgmap v62: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: pgmap v62: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: pgmap v62: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:36.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:36.673 DEBUG:teuthology.orchestra.run.smithi156:osd.1> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.1.service 2023-11-16T23:53:36.676 INFO:tasks.cephadm:Deploying osd.2 on smithi173 with /dev/vg_nvme/lv_4... 2023-11-16T23:53:36.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T23:53:36.704 INFO:journalctl@ceph.osd.1.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:53:37.321 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:53:37.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:37 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:37.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:37 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:53:37.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:37 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:37.418 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:53:37 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-16T23:53:37.245+0000 7fddd5af3640 -1 osd.1 0 log_to_monitors true 2023-11-16T23:53:37.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:37 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:37 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:53:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:37 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:37 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:37 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:53:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:37 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:38.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:38 smithi173 ceph-mon[131534]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:38.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:38 smithi173 ceph-mon[131534]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:38 smithi173 ceph-mon[131534]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:38 smithi023 ceph-mon[97824]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:38 smithi023 ceph-mon[97824]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:38 smithi023 ceph-mon[97824]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.522 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:38 smithi156 ceph-mon[131105]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:38.522 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:38 smithi156 ceph-mon[131105]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.522 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:38 smithi156 ceph-mon[131105]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T23:53:38.918 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T23:53:38.918 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T23:53:38.918 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2023-11-16T23:53:38.919 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2023-11-16T23:53:38.919 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127846 s, 820 MB/s 2023-11-16T23:53:38.919 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2023-11-16T23:53:39.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2023-11-16T23:53:39.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T23:53:39.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.608 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-16T23:53:39.273+0000 7fddcb16a700 -1 osd.1 0 waiting for initial osdmap 2023-11-16T23:53:39.608 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-16T23:53:39.277+0000 7fddc3758700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:39.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:39.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:39.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:39.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:39.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:39.992 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:53:40.536 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:53:40.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:53:40.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi156 2023-11-16T23:53:40.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: pgmap v65: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:40.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi156 to 17168M 2023-11-16T23:53:40.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:40.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi156 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: pgmap v65: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi156 to 17168M 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T23:53:40.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:40.612 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi156 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: pgmap v65: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi156 to 17168M 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:53:40.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:40.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:53:40.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T23:53:40.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:40.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 -- 172.21.15.173:0/690651673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 msgr2=0x7fa04c100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 --2- 172.21.15.173:0/690651673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c100770 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa040009990 tx=0x7fa040009ca0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 -- 172.21.15.173:0/690651673 shutdown_connections 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 --2- 172.21.15.173:0/690651673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101ef0 0x7fa04c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 --2- 172.21.15.173:0/690651673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c100770 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 --2- 172.21.15.173:0/690651673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa04c101560 0x7fa04c1019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 -- 172.21.15.173:0/690651673 >> 172.21.15.173:0/690651673 conn(0x7fa04c0fb970 msgr2=0x7fa04c0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 -- 172.21.15.173:0/690651673 shutdown_connections 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.316+0000 7fa0512bc700 1 -- 172.21.15.173:0/690651673 wait complete. 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 Processor -- start 2023-11-16T23:53:41.321 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 -- start start 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101560 0x7fa04c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa04c101ef0 0x7fa04c19d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c19dd20 con 0x7fa04c101ef0 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c19de60 con 0x7fa04c101560 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa0512bc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c19dff0 con 0x7fa04c100360 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51114/0 (socket says 172.21.15.173:51114) 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 -- 172.21.15.173:0/2889762472 learned_addr learned my addr 172.21.15.173:0/2889762472 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 -- 172.21.15.173:0/2889762472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101560 msgr2=0x7fa04c1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04a7fc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101560 0x7fa04c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:41.322 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101560 0x7fa04c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 -- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa04c101ef0 msgr2=0x7fa04c19d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.317+0000 7fa04affd700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa04c101ef0 0x7fa04c19d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa04affd700 1 -- 172.21.15.173:0/2889762472 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0400095f0 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa04affd700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa04000f690 tx=0x7fa04000f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04001d070 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa040004cf0 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040005480 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c10bd80 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.318+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c10bbf0 con 0x7fa04c100360 2023-11-16T23:53:41.323 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.319+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c107a00 con 0x7fa04c100360 2023-11-16T23:53:41.324 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.320+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fa04000fba0 con 0x7fa04c100360 2023-11-16T23:53:41.324 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.320+0000 7fa033fff700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 0x7fa0340a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:53:41.324 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.320+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fa0400cc2d0 con 0x7fa04c100360 2023-11-16T23:53:41.324 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.321+0000 7fa04a7fc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 0x7fa0340a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:53:41.324 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.321+0000 7fa04a7fc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 0x7fa0340a35f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:53:41.326 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.325+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa04005b730 con 0x7fa04c100360 2023-11-16T23:53:41.520 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:41.518+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa04c061c00 con 0x7fa0340a1140 2023-11-16T23:53:41.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:41 smithi156 ceph-mon[131105]: osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] boot 2023-11-16T23:53:41.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:41 smithi156 ceph-mon[131105]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T23:53:41.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:41 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:41.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:41 smithi173 ceph-mon[131534]: osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] boot 2023-11-16T23:53:41.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:41 smithi173 ceph-mon[131534]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T23:53:41.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:41 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:41.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:41 smithi023 ceph-mon[97824]: osd.1 [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] boot 2023-11-16T23:53:41.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:41 smithi023 ceph-mon[97824]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T23:53:41.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:41 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T23:53:42.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: pgmap v68: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:42.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: pgmap v68: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: pgmap v68: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:53:42.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:43.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:43 smithi173 ceph-mon[131534]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:43.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:43 smithi156 ceph-mon[131105]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:43.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:43 smithi023 ceph-mon[97824]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:44 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/1012474039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:44 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:44 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]': finished 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:44 smithi156 ceph-mon[131105]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:44 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/1012474039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:44 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:44 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]': finished 2023-11-16T23:53:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:44 smithi173 ceph-mon[131534]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T23:53:44.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:53:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:44 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/1012474039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:44 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]: dispatch 2023-11-16T23:53:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:44 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b22c974-66e4-455c-882c-4804d393aec2"}]': finished 2023-11-16T23:53:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:44 smithi023 ceph-mon[97824]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T23:53:44.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:53:45.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:45 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/455465704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:45.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:45 smithi173 ceph-mon[131534]: pgmap v72: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:45.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:45 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/455465704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:45.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:45 smithi156 ceph-mon[131105]: pgmap v72: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:45.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:45 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/455465704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:53:45.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:45 smithi023 ceph-mon[97824]: pgmap v72: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:48 smithi173 ceph-mon[131534]: pgmap v73: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:48.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:48 smithi156 ceph-mon[131105]: pgmap v73: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:48.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:48 smithi023 ceph-mon[97824]: pgmap v73: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:49.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:49 smithi173 ceph-mon[131534]: pgmap v74: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:49.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:49 smithi156 ceph-mon[131105]: pgmap v74: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:49.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:49 smithi023 ceph-mon[97824]: pgmap v74: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:51.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T23:53:51.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T23:53:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:51.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T23:53:51.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:51 smithi173 ceph-mon[131534]: Deploying daemon osd.2 on smithi173 2023-11-16T23:53:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:51 smithi173 ceph-mon[131534]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:52.106 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:51 smithi156 ceph-mon[131105]: Deploying daemon osd.2 on smithi173 2023-11-16T23:53:52.106 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:51 smithi156 ceph-mon[131105]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:52.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:51 smithi023 ceph-mon[97824]: Deploying daemon osd.2 on smithi173 2023-11-16T23:53:52.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:51 smithi023 ceph-mon[97824]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:54.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:54 smithi173 ceph-mon[131534]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:54.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:54 smithi156 ceph-mon[131105]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:54 smithi023 ceph-mon[97824]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:56.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:56 smithi173 ceph-mon[131534]: pgmap v77: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:56.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:56.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:56 smithi156 ceph-mon[131105]: pgmap v77: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:56.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:56.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:56 smithi023 ceph-mon[97824]: pgmap v77: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:58.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:58.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:58.308 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:58.308 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:53:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:53:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:53:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:58.590 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.589+0000 7fa033fff700 1 -- 172.21.15.173:0/2889762472 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa04c061c00 con 0x7fa0340a1140 2023-11-16T23:53:58.590 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 2 on host 'smithi173' 2023-11-16T23:53:58.592 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.591+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 msgr2=0x7fa0340a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:58.592 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.591+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 0x7fa0340a35f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c008040 comp rx=0 tx=0).stop 2023-11-16T23:53:58.592 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.591+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 msgr2=0x7fa04c196190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa04000f690 tx=0x7fa04000f6c0 comp rx=0 tx=0).stop 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 shutdown_connections 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa04c101ef0 0x7fa04c19d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa0340a1140 0x7fa0340a35f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa04c100360 0x7fa04c196190 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 --2- 172.21.15.173:0/2889762472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa04c101560 0x7fa04c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 >> 172.21.15.173:0/2889762472 conn(0x7fa04c0fb970 msgr2=0x7fa04c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 shutdown_connections 2023-11-16T23:53:58.593 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:53:58.592+0000 7fa0512bc700 1 -- 172.21.15.173:0/2889762472 wait complete. 2023-11-16T23:53:59.073 DEBUG:teuthology.orchestra.run.smithi173:osd.2> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.2.service 2023-11-16T23:53:59.075 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2023-11-16T23:53:59.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T23:53:59.108 INFO:journalctl@ceph.osd.2.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:53:59.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:53:59.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:53:59 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:59.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:53:59.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:53:59.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:53:59.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:53:59 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:53:59.628 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:53:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-16T23:53:59.370+0000 7f3cd20a8640 -1 osd.2 0 log_to_monitors true 2023-11-16T23:53:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:54:00.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:00 smithi023 ceph-mon[97824]: pgmap v79: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:00.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:00 smithi023 ceph-mon[97824]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:00.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:00 smithi023 ceph-mon[97824]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:00 smithi156 ceph-mon[131105]: pgmap v79: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:00 smithi156 ceph-mon[131105]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:00 smithi156 ceph-mon[131105]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:00 smithi173 ceph-mon[131534]: pgmap v79: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:00 smithi173 ceph-mon[131534]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:00.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:00 smithi173 ceph-mon[131534]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T23:54:01.356 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-16T23:54:01.086+0000 7f3cc771f700 -1 osd.2 0 waiting for initial osdmap 2023-11-16T23:54:01.356 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-16T23:54:01.090+0000 7f3cc1d11700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-mon[131534]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-mon[131534]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-mon[131534]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:01 smithi173 ceph-mon[131534]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:01 smithi156 ceph-mon[131105]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T23:54:01.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:01 smithi156 ceph-mon[131105]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T23:54:01.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:01 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:01.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:01 smithi156 ceph-mon[131105]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:01 smithi156 ceph-mon[131105]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:01 smithi023 ceph-mon[97824]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T23:54:01.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:01 smithi023 ceph-mon[97824]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T23:54:01.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:01 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:01.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:01 smithi023 ceph-mon[97824]: from='osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:01 smithi023 ceph-mon[97824]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:54:01.811 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T23:54:01.812 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T23:54:01.812 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2023-11-16T23:54:01.812 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2023-11-16T23:54:01.812 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0310511 s, 338 MB/s 2023-11-16T23:54:01.812 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2023-11-16T23:54:02.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2023-11-16T23:54:02.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: pgmap v81: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: pgmap v81: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:02.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:02.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:02.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: pgmap v81: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:02.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:02.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:02.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:03.085 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:54:03.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi173 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi173 to 17168M 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] boot 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T23:54:03.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:03 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:04.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:54:04.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi173 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi173 to 17168M 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] boot 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T23:54:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:03 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi173 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi173 to 17168M 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: osd.2 [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] boot 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T23:54:04.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:03 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 -- 172.21.15.23:0/3569836075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1014f0 msgr2=0x7fd1bc101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 --2- 172.21.15.23:0/3569836075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc101940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009990 tx=0x7fd1b0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 -- 172.21.15.23:0/3569836075 shutdown_connections 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 --2- 172.21.15.23:0/3569836075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 0x7fd1bc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 --2- 172.21.15.23:0/3569836075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 --2- 172.21.15.23:0/3569836075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc101940 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 -- 172.21.15.23:0/3569836075 >> 172.21.15.23:0/3569836075 conn(0x7fd1bc0fb900 msgr2=0x7fd1bc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:54:04.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 -- 172.21.15.23:0/3569836075 shutdown_connections 2023-11-16T23:54:04.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.469+0000 7fd1c2419700 1 -- 172.21.15.23:0/3569836075 wait complete. 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.470+0000 7fd1c2419700 1 Processor -- start 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.470+0000 7fd1c2419700 1 -- start start 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.470+0000 7fd1c2419700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.470+0000 7fd1c2419700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.470+0000 7fd1c2419700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 0x7fd1bc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1c2419700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc19fe10 con 0x7fd1bc1002f0 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1c2419700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc19ff50 con 0x7fd1bc1014f0 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1c2419700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc1a00e0 con 0x7fd1bc101e80 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:04.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47240/0 (socket says 172.21.15.23:47240) 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 -- 172.21.15.23:0/2791187515 learned_addr learned my addr 172.21.15.23:0/2791187515 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bb7fe700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1c09b6700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 0x7fd1bc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 -- 172.21.15.23:0/2791187515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 msgr2=0x7fd1bc19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 0x7fd1bc19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 -- 172.21.15.23:0/2791187515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1014f0 msgr2=0x7fd1bc1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.471+0000 7fd1bbfff700 1 -- 172.21.15.23:0/2791187515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00095f0 con 0x7fd1bc1002f0 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1bbfff700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00d9b0 tx=0x7fd1ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:54:04.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac00b950 con 0x7fd1bc1002f0 2023-11-16T23:54:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1ac007620 con 0x7fd1bc1002f0 2023-11-16T23:54:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc10bcd0 con 0x7fd1bc1002f0 2023-11-16T23:54:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac00f3c0 con 0x7fd1bc1002f0 2023-11-16T23:54:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.472+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc10bb40 con 0x7fd1bc1002f0 2023-11-16T23:54:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.473+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc066ea0 con 0x7fd1bc1002f0 2023-11-16T23:54:04.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.475+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fd1ac010460 con 0x7fd1bc1002f0 2023-11-16T23:54:04.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.476+0000 7fd1b97fa700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 0x7fd1a40a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.477+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7fd1ac0cbb90 con 0x7fd1bc1002f0 2023-11-16T23:54:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.477+0000 7fd1bb7fe700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 0x7fd1a40a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.477+0000 7fd1bb7fe700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 0x7fd1a40a3570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000b2e0 tx=0x7fd1b0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:54:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.479+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd1ac096700 con 0x7fd1bc1002f0 2023-11-16T23:54:04.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:04.714+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1bc107e90 con 0x7fd1a40a10c0 2023-11-16T23:54:05.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:04 smithi023 ceph-mon[97824]: pgmap v84: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:04 smithi023 ceph-mon[97824]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T23:54:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:04 smithi173 ceph-mon[131534]: pgmap v84: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:04 smithi173 ceph-mon[131534]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T23:54:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:04 smithi156 ceph-mon[131105]: pgmap v84: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T23:54:05.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:05.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:04 smithi156 ceph-mon[131105]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T23:54:05.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T23:54:05.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113060]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:05.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113060]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:05.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113060]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:05.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113060]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:05.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:05.636+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd1ac096ad0 con 0x7fd1bc1002f0 2023-11-16T23:54:05.945 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113104]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:05.946 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:05.946 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113104]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:05.946 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 16 23:54:05 smithi023 sudo[113104]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:05.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:05.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T23:54:05.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:05.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:05.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:05.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:05.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:05 smithi023 ceph-mon[97824]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:06.079 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:06.079 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T23:54:06.079 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:06.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:06.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:06.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:06.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:05 smithi156 ceph-mon[131105]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:06.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T23:54:06.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T23:54:06.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:06.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:06.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:06.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:06.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:05 smithi173 ceph-mon[131534]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:06.331 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138507]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:06.331 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138507]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:06.332 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138507]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:06.332 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138507]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:06.606 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138510]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:06.607 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:06.607 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138510]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:06.607 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 16 23:54:06 smithi156 sudo[138510]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:06.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:06 smithi023 ceph-mon[97824]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T23:54:06.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:06 smithi023 ceph-mon[97824]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T23:54:06.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:06.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:06.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:06 smithi173 ceph-mon[131534]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T23:54:06.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:06 smithi173 ceph-mon[131534]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T23:54:06.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:06.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:06.852 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139572]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:06.852 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139572]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:06.852 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139572]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:06.852 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139572]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:07.106 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139575]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:07.107 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139575]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:07.107 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139575]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:07.107 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 16 23:54:06 smithi173 sudo[139575]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:07.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:06 smithi156 ceph-mon[131105]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T23:54:07.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:06 smithi156 ceph-mon[131105]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T23:54:07.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:07.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113220]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113220]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113220]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:07.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113220]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:08.232 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:07 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:08.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:07 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:08.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139578]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:08.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:08.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139578]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:08.233 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139578]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:08.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113247]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:08.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113247]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:08.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113247]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:08.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 sudo[113247]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:08.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:08.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:07 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:08.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:07 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:08.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:07 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:08.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139581]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:08.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139581]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:08.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139581]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 sudo[139581]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:08.969 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138513]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T23:54:08.969 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:08.969 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138513]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:08.969 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138513]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:09.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:09.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/733734545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]': finished 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:08 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4218260289' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:09.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138516]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138516]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138516]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 sudo[138516]: pam_unix(sudo:session): session closed for user root 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/733734545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]': finished 2023-11-16T23:54:09.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:08 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4218260289' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:09.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/733734545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82122923-cabb-475e-ab26-d6a7bb6c88ad"}]': finished 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:09.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T23:54:09.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T23:54:09.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T23:54:09.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:08 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4218260289' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:10.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:09 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:10.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:09 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:10.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:09 smithi023 ceph-mon[97824]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:10.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:09 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:10.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:09 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:10.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:09 smithi173 ceph-mon[131534]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:10.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:09 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T23:54:10.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:09 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T23:54:10.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:09 smithi156 ceph-mon[131105]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:12.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:12 smithi023 ceph-mon[97824]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:12.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:12 smithi173 ceph-mon[131534]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:12 smithi156 ceph-mon[131105]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:14.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:14 smithi023 ceph-mon[97824]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-11-16T23:54:14.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:14 smithi173 ceph-mon[131534]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-11-16T23:54:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:14 smithi156 ceph-mon[131105]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-11-16T23:54:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:16 smithi173 ceph-mon[131534]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-11-16T23:54:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T23:54:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:16 smithi156 ceph-mon[131105]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-11-16T23:54:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T23:54:16.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:16 smithi023 ceph-mon[97824]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-11-16T23:54:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T23:54:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:17.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:17 smithi023 ceph-mon[97824]: Deploying daemon osd.3 on smithi023 2023-11-16T23:54:17.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:17 smithi173 ceph-mon[131534]: Deploying daemon osd.3 on smithi023 2023-11-16T23:54:17.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:17 smithi156 ceph-mon[131105]: Deploying daemon osd.3 on smithi023 2023-11-16T23:54:18.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:18 smithi173 ceph-mon[131534]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-16T23:54:18.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:18 smithi156 ceph-mon[131105]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-16T23:54:18.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:18 smithi023 ceph-mon[97824]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-16T23:54:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:19 smithi023 ceph-mon[97824]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-11-16T23:54:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:19 smithi173 ceph-mon[131534]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-11-16T23:54:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:19 smithi156 ceph-mon[131105]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-11-16T23:54:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:22 smithi023 ceph-mon[97824]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:22 smithi173 ceph-mon[131534]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:22.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:22 smithi156 ceph-mon[131105]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:22.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:22.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:23.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:23.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:23.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:24.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:24 smithi023 ceph-mon[97824]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:24.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:24 smithi173 ceph-mon[131534]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:24 smithi156 ceph-mon[131105]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-16T23:54:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.920+0000 7fd1b97fa700 1 -- 172.21.15.23:0/2791187515 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd1bc107e90 con 0x7fd1a40a10c0 2023-11-16T23:54:35.357 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 msgr2=0x7fd1a40a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 0x7fd1a40a3570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000b2e0 tx=0x7fd1b0005fb0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 msgr2=0x7fd1bc198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00d9b0 tx=0x7fd1ac00dcc0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 shutdown_connections 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd1a40a10c0 0x7fd1a40a3570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.922+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1bc101e80 0x7fd1bc19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.923+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1bc1002f0 0x7fd1bc198280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.923+0000 7fd1c2419700 1 --2- 172.21.15.23:0/2791187515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1bc1014f0 0x7fd1bc1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.923+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 >> 172.21.15.23:0/2791187515 conn(0x7fd1bc0fb900 msgr2=0x7fd1bc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:54:35.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.923+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 shutdown_connections 2023-11-16T23:54:35.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:54:25.923+0000 7fd1c2419700 1 -- 172.21.15.23:0/2791187515 wait complete. 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.359 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.360 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:35.362 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.3.service 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:26 smithi023 ceph-mon[97824]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:26 smithi023 ceph-mon[97824]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:27 smithi023 ceph-mon[97824]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:54:35.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] boot 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:29 smithi023 ceph-mon[97824]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi023 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi023 to 8635M 2023-11-16T23:54:35.402 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:30 smithi023 ceph-mon[97824]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:31 smithi023 ceph-mon[97824]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:31 smithi023 ceph-mon[97824]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:33 smithi023 ceph-mon[97824]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T23:54:35.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:33 smithi023 ceph-mon[97824]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:54:35.405 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.406 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:26 smithi156 ceph-mon[131105]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:26 smithi156 ceph-mon[131105]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:27 smithi156 ceph-mon[131105]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] boot 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:29 smithi156 ceph-mon[131105]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi023 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi023 to 8635M 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:30 smithi156 ceph-mon[131105]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:31 smithi156 ceph-mon[131105]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:31 smithi156 ceph-mon[131105]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:33 smithi156 ceph-mon[131105]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:33 smithi156 ceph-mon[131105]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.410 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:26 smithi173 ceph-mon[131534]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:26 smithi173 ceph-mon[131534]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T23:54:35.411 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: from='osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:27 smithi173 ceph-mon[131534]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: osd.3 [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] boot 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T23:54:35.412 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:35.413 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:29 smithi173 ceph-mon[131534]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi023 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi023 to 8635M 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:30 smithi173 ceph-mon[131534]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:31 smithi173 ceph-mon[131534]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:31 smithi173 ceph-mon[131534]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:33 smithi173 ceph-mon[131534]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T23:54:35.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:33 smithi173 ceph-mon[131534]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:54:35.415 INFO:tasks.cephadm:Deploying osd.4 on smithi156 with /dev/vg_nvme/lv_3... 2023-11-16T23:54:35.415 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T23:54:35.489 INFO:journalctl@ceph.osd.3.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:54:36.064 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:54:36.323 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:36 smithi156 ceph-mon[131105]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:54:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:36 smithi173 ceph-mon[131534]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:54:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:36 smithi023 ceph-mon[97824]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:54:37.569 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T23:54:37.570 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T23:54:37.570 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-11-16T23:54:37.570 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-11-16T23:54:37.570 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00815352 s, 1.3 GB/s 2023-11-16T23:54:37.570 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-11-16T23:54:38.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2023-11-16T23:54:38.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:38 smithi156 ceph-mon[131105]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T23:54:38.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:38 smithi173 ceph-mon[131534]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T23:54:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:38 smithi023 ceph-mon[97824]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-11-16T23:54:38.647 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:54:39.674 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:39 smithi156 ceph-mon[131105]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:54:39.882 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 -- 172.21.15.156:0/1837072974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 msgr2=0x7ff33c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:39.882 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 --2- 172.21.15.156:0/1837072974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c107710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff33400b0a0 tx=0x7ff33400b3b0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.882 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 -- 172.21.15.156:0/1837072974 shutdown_connections 2023-11-16T23:54:39.882 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 --2- 172.21.15.156:0/1837072974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c107710 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.883 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 --2- 172.21.15.156:0/1837072974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 0x7ff33c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.883 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 --2- 172.21.15.156:0/1837072974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c0ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.883 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 -- 172.21.15.156:0/1837072974 >> 172.21.15.156:0/1837072974 conn(0x7ff33c0f9f70 msgr2=0x7ff33c0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:54:39.883 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 -- 172.21.15.156:0/1837072974 shutdown_connections 2023-11-16T23:54:39.883 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.881+0000 7ff343744700 1 -- 172.21.15.156:0/1837072974 wait complete. 2023-11-16T23:54:39.887 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 Processor -- start 2023-11-16T23:54:39.887 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 -- start start 2023-11-16T23:54:39.887 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:39.887 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:39.887 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 0x7ff33c19d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c19dca0 con 0x7ff33c0ff4b0 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c19dde0 con 0x7ff33c0ffe40 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff343744700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c19df70 con 0x7ff33c100770 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff340cdf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff340cdf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35272/0 (socket says 172.21.15.156:35272) 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff340cdf700 1 -- 172.21.15.156:0/1940997205 learned_addr learned my addr 172.21.15.156:0/1940997205 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff341ce1700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 0x7ff33c19d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.882+0000 7ff3414e0700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 -- 172.21.15.156:0/1940997205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 msgr2=0x7ff33c19d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 0x7ff33c19d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 -- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 msgr2=0x7ff33c196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:39.888 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 -- 172.21.15.156:0/1940997205 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff334009d00 con 0x7ff33c0ffe40 2023-11-16T23:54:39.889 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff3414e0700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:54:39.889 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff340cdf700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:54:39.889 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338009790 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c10ab60 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff33800cbd0 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338018680 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.883+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c10a9d0 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.884+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c066ea0 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.885+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff33800cd40 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.886+0000 7ff3327fc700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 0x7ff3280a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.886+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7ff338014070 con 0x7ff33c0ffe40 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.886+0000 7ff3414e0700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 0x7ff3280a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:54:39.890 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.887+0000 7ff3414e0700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 0x7ff3280a7a10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff32c005950 tx=0x7ff32c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:54:39.891 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:39.889+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff338096590 con 0x7ff33c0ffe40 2023-11-16T23:54:40.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:39 smithi023 ceph-mon[97824]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:54:40.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:40.082+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff33c0feb70 con 0x7ff3280a5560 2023-11-16T23:54:40.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:39 smithi173 ceph-mon[131534]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:54:40.971 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:40 smithi156 ceph-mon[131105]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:40.971 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:40.971 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:40.971 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:40 smithi023 ceph-mon[97824]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:41.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:40 smithi173 ceph-mon[131534]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:54:41.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:54:41.110 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:54:41.110 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:41.828 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:41 smithi156 ceph-mon[131105]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:42.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:41 smithi023 ceph-mon[97824]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:42.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:41 smithi173 ceph-mon[131534]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:43.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:42 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/2946781884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:42 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:42 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]': finished 2023-11-16T23:54:43.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:42 smithi023 ceph-mon[97824]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T23:54:43.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:43.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:42 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/2946781884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:42 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:42 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]': finished 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:42 smithi173 ceph-mon[131534]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:42 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/2946781884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:42 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]: dispatch 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:42 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbe06115-7a9f-4be5-b2df-3ac182c2f34f"}]': finished 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:42 smithi156 ceph-mon[131105]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T23:54:43.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:44.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:43 smithi023 ceph-mon[97824]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:44.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:43 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/3901020607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:44.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:43 smithi173 ceph-mon[131534]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:44.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:43 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/3901020607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:44.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:43 smithi156 ceph-mon[131105]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:44.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:43 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/3901020607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:54:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:46 smithi173 ceph-mon[131534]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:46.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:46 smithi156 ceph-mon[131105]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:46.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:46 smithi023 ceph-mon[97824]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:48 smithi173 ceph-mon[131534]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:48.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:48 smithi156 ceph-mon[131105]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:48.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:48 smithi023 ceph-mon[97824]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:50.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:49 smithi023 ceph-mon[97824]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T23:54:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:50.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:49 smithi156 ceph-mon[131105]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:50.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T23:54:50.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:49 smithi173 ceph-mon[131534]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T23:54:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:50.856 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:50 smithi156 ceph-mon[131105]: Deploying daemon osd.4 on smithi156 2023-11-16T23:54:51.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:50 smithi023 ceph-mon[97824]: Deploying daemon osd.4 on smithi156 2023-11-16T23:54:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:50 smithi173 ceph-mon[131534]: Deploying daemon osd.4 on smithi156 2023-11-16T23:54:51.968 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:51 smithi156 ceph-mon[131105]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:52.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:51 smithi023 ceph-mon[97824]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:52.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:51 smithi173 ceph-mon[131534]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:54.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:54 smithi156 ceph-mon[131105]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:54.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:54.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:54 smithi173 ceph-mon[131534]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:54.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:54.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:54 smithi023 ceph-mon[97824]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:56.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:56 smithi156 ceph-mon[131105]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:56.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:56 smithi173 ceph-mon[131534]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:56.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:56 smithi023 ceph-mon[97824]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:56.952 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 4 on host 'smithi156' 2023-11-16T23:54:56.952 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.950+0000 7ff3327fc700 1 -- 172.21.15.156:0/1940997205 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff33c0feb70 con 0x7ff3280a5560 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 msgr2=0x7ff3280a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 0x7ff3280a7a10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff32c005950 tx=0x7ff32c0058e0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 msgr2=0x7ff33c196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800ecb0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 shutdown_connections 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff3280a5560 0x7ff3280a7a10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff33c100770 0x7ff33c19d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.954 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0ff4b0 0x7ff33c196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.955 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 --2- 172.21.15.156:0/1940997205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff33c0ffe40 0x7ff33c196650 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:54:56.955 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 >> 172.21.15.156:0/1940997205 conn(0x7ff33c0f9f70 msgr2=0x7ff33c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:54:56.955 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 shutdown_connections 2023-11-16T23:54:56.955 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:54:56.953+0000 7ff343744700 1 -- 172.21.15.156:0/1940997205 wait complete. 2023-11-16T23:54:57.426 DEBUG:teuthology.orchestra.run.smithi156:osd.4> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.4.service 2023-11-16T23:54:57.428 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2023-11-16T23:54:57.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T23:54:57.471 INFO:journalctl@ceph.osd.4.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:54:57.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:57 smithi173 ceph-mon[131534]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:54:57.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:57.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-mon[131105]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:54:57.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:54:57.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:57 smithi023 ceph-mon[97824]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:58.090 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:54:58.107 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 16 23:54:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-16T23:54:57.749+0000 7fb03b422640 -1 osd.4 0 log_to_monitors true 2023-11-16T23:54:58.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:58 smithi156 ceph-mon[131105]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:58.485 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:58 smithi156 ceph-mon[131105]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:58 smithi173 ceph-mon[131534]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:58 smithi173 ceph-mon[131534]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:58.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:58 smithi023 ceph-mon[97824]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:58.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:58 smithi023 ceph-mon[97824]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T23:54:59.574 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T23:54:59.574 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T23:54:59.575 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2023-11-16T23:54:59.575 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2023-11-16T23:54:59.575 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110227 s, 951 MB/s 2023-11-16T23:54:59.575 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2023-11-16T23:54:59.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T23:54:59.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T23:54:59.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:59.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:54:59 smithi173 ceph-mon[131534]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:59.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-16T23:54:59.344+0000 7fb030a99700 -1 osd.4 0 waiting for initial osdmap 2023-11-16T23:54:59.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-16T23:54:59.353+0000 7fb02a88a700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:54:59.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T23:54:59.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T23:54:59.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:59.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:54:59 smithi156 ceph-mon[131105]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:54:59.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T23:54:59.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T23:54:59.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:54:59.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: from='osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:54:59.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:54:59 smithi023 ceph-mon[97824]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:55:00.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2023-11-16T23:55:00.670 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi156 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi156 to 8584M 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:00.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:00.770 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:00.770 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.770 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:55:00.770 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:00 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:00.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi156 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi156 to 8584M 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:00.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:00 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:00.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi156 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi156 to 8584M 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:55:00.859 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:00 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:01.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:55:01.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] boot 2023-11-16T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:01 smithi173 ceph-mon[131534]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:55:01.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] boot 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:01 smithi023 ceph-mon[97824]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:55:01.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:55:01.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:55:01.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: osd.4 [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] boot 2023-11-16T23:55:01.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T23:55:01.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T23:55:01.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:01 smithi156 ceph-mon[131105]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-11-16T23:55:01.885 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.883+0000 7f433552f700 1 -- 172.21.15.173:0/3791583239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 msgr2=0x7f43301088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.883+0000 7f433552f700 1 --2- 172.21.15.173:0/3791583239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f43301088e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f432800b0a0 tx=0x7f432800b3b0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 -- 172.21.15.173:0/3791583239 shutdown_connections 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- 172.21.15.173:0/3791583239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f43301088e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- 172.21.15.173:0/3791583239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 0x7f43300fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- 172.21.15.173:0/3791583239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 0x7f4330101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 -- 172.21.15.173:0/3791583239 >> 172.21.15.173:0/3791583239 conn(0x7f43300fa000 msgr2=0x7f43300fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 -- 172.21.15.173:0/3791583239 shutdown_connections 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 -- 172.21.15.173:0/3791583239 wait complete. 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 Processor -- start 2023-11-16T23:55:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 -- start start 2023-11-16T23:55:01.888 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 0x7f43301960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:01.888 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 0x7f4330196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:01.888 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.884+0000 7f433552f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:01.888 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f433552f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019dc70 con 0x7f4330101690 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f433552f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019ddb0 con 0x7f43300fe610 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f433552f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019df40 con 0x7f4330102020 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51202/0 (socket says 172.21.15.173:51202) 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 -- 172.21.15.173:0/1336216152 learned_addr learned my addr 172.21.15.173:0/1336216152 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432e7fc700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 0x7f4330196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432effd700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 0x7f43301960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 -- 172.21.15.173:0/1336216152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 msgr2=0x7f43301960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 0x7f43301960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 -- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 msgr2=0x7f4330196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 0x7f4330196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:01.889 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 -- 172.21.15.173:0/1336216152 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328009d00 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f432f7fe700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4328004d00 tx=0x7f4328004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432800e040 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433010bd20 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f432801bd20 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328005920 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.885+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433010bb90 con 0x7f4330102020 2023-11-16T23:55:01.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.886+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4330107900 con 0x7f4330102020 2023-11-16T23:55:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.887+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4328005040 con 0x7f4330102020 2023-11-16T23:55:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.888+0000 7f4317fff700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 0x7f43180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.888+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f43280cda00 con 0x7f4330102020 2023-11-16T23:55:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.888+0000 7f432effd700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 0x7f43180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:01.891 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.889+0000 7f432effd700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 0x7f43180a3600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4330068e40 tx=0x7f4320005d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:01.891+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4328098250 con 0x7f4330102020 2023-11-16T23:55:02.084 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:02.083+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4330061c70 con 0x7f43180a1150 2023-11-16T23:55:02.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:02 smithi023 ceph-mon[97824]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T23:55:02.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:02 smithi023 ceph-mon[97824]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:02.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:02.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:02.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:02.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:02 smithi173 ceph-mon[131534]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T23:55:02.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:02 smithi173 ceph-mon[131534]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:02.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:02 smithi156 ceph-mon[131105]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:02 smithi156 ceph-mon[131105]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:02.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:03.551 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:03 smithi173 ceph-mon[131534]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:03.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:03 smithi023 ceph-mon[97824]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:03.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:03 smithi156 ceph-mon[131105]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:05.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:04 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/3748473139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:04 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:04 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]': finished 2023-11-16T23:55:05.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:04 smithi023 ceph-mon[97824]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T23:55:05.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:05.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:04 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/3748473139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:04 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:04 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]': finished 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:04 smithi173 ceph-mon[131534]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:04 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/3748473139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:04 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]: dispatch 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:04 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cb74da1-90da-4bfa-a528-b08833cabdcb"}]': finished 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:04 smithi156 ceph-mon[131105]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T23:55:05.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:06.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:05 smithi023 ceph-mon[97824]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:06.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:05 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/408562280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:06.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:05 smithi173 ceph-mon[131534]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:06.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:05 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/408562280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:06.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:05 smithi156 ceph-mon[131105]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:06.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:05 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/408562280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:08.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:08 smithi173 ceph-mon[131534]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:08.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:08 smithi156 ceph-mon[131105]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:08.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:08 smithi023 ceph-mon[97824]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:09.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:09 smithi173 ceph-mon[131534]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:10.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:09 smithi023 ceph-mon[97824]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:10.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:09 smithi156 ceph-mon[131105]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:12.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:12 smithi173 ceph-mon[131534]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:12.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T23:55:12.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:12 smithi156 ceph-mon[131105]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T23:55:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:12.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:12 smithi023 ceph-mon[97824]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:12.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T23:55:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:13.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:13 smithi173 ceph-mon[131534]: Deploying daemon osd.5 on smithi173 2023-11-16T23:55:13.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:13 smithi156 ceph-mon[131105]: Deploying daemon osd.5 on smithi173 2023-11-16T23:55:13.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:13 smithi023 ceph-mon[97824]: Deploying daemon osd.5 on smithi173 2023-11-16T23:55:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:14 smithi156 ceph-mon[131105]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:14.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:14 smithi173 ceph-mon[131534]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:14.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:14 smithi023 ceph-mon[97824]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:16.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:16 smithi173 ceph-mon[131534]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:16 smithi156 ceph-mon[131105]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:16 smithi023 ceph-mon[97824]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:18.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:18 smithi173 ceph-mon[131534]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:18.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:18 smithi156 ceph-mon[131105]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:18.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:18 smithi023 ceph-mon[97824]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:19.051 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2023-11-16T23:55:19.052 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.051+0000 7f4317fff700 1 -- 172.21.15.173:0/1336216152 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4330061c70 con 0x7f43180a1150 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 msgr2=0x7f43180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 0x7f43180a3600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4330068e40 tx=0x7f4320005d70 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 msgr2=0x7f433019d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4328004d00 tx=0x7f4328004de0 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 shutdown_connections 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43180a1150 0x7f43180a3600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4330102020 0x7f433019d6a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43300fe610 0x7f43301960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 --2- 172.21.15.173:0/1336216152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4330101690 0x7f4330196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:19.054 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 >> 172.21.15.173:0/1336216152 conn(0x7f43300fa000 msgr2=0x7f4330104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:55:19.055 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 shutdown_connections 2023-11-16T23:55:19.055 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:55:19.053+0000 7f433552f700 1 -- 172.21.15.173:0/1336216152 wait complete. 2023-11-16T23:55:19.458 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.5.service 2023-11-16T23:55:19.460 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2023-11-16T23:55:19.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:55:19.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:19 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:19.493 INFO:journalctl@ceph.osd.5.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:55:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:19.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:19 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:19.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:19 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:55:20.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:20 smithi173 ceph-mon[131534]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:20.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:20 smithi173 ceph-mon[131534]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:20.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:20 smithi173 ceph-mon[131534]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:20.352 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 16 23:55:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-16T23:55:20.045+0000 7f53d9030640 -1 osd.5 0 log_to_monitors true 2023-11-16T23:55:20.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:20 smithi023 ceph-mon[97824]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:20 smithi023 ceph-mon[97824]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:20 smithi023 ceph-mon[97824]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:20.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:20 smithi156 ceph-mon[131105]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:20.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:20 smithi156 ceph-mon[131105]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:20.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:20 smithi156 ceph-mon[131105]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T23:55:21.857 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-16T23:55:21.654+0000 7f53ce6a7700 -1 osd.5 0 waiting for initial osdmap 2023-11-16T23:55:21.857 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-16T23:55:21.661+0000 7f53c6494700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:21 smithi173 ceph-mon[131534]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:22.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T23:55:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T23:55:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:21 smithi023 ceph-mon[97824]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:22.106 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T23:55:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T23:55:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: from='osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:55:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:21 smithi156 ceph-mon[131105]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T23:55:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T23:55:22.338 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2023-11-16T23:55:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2023-11-16T23:55:22.339 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0312264 s, 336 MB/s 2023-11-16T23:55:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2023-11-16T23:55:22.930 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:55:22.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T23:55:22.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:22.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:22.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi173 2023-11-16T23:55:22.931 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi173 to 8584M 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:22.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:22.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:22.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:22.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:22.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:55:22.933 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:22 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:22.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2023-11-16T23:55:23.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi173 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi173 to 8584M 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:22 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:23.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi173 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi173 to 8584M 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:55:23.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:22 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:23.804 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:55:24.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] boot 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:23 smithi023 ceph-mon[97824]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T23:55:24.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:55:24.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] boot 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:23 smithi173 ceph-mon[131534]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:55:24.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: osd.5 [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] boot 2023-11-16T23:55:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T23:55:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T23:55:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-11-16T23:55:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:23 smithi156 ceph-mon[131105]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.439+0000 7fb579a21700 1 -- 172.21.15.23:0/4180575003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 msgr2=0x7fb574108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 --2- 172.21.15.23:0/4180575003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb574108740 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00b0a0 tx=0x7fb56c00b3b0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 -- 172.21.15.23:0/4180575003 shutdown_connections 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 --2- 172.21.15.23:0/4180575003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb574108740 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 --2- 172.21.15.23:0/4180575003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 0x7fb574100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 --2- 172.21.15.23:0/4180575003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 -- 172.21.15.23:0/4180575003 >> 172.21.15.23:0/4180575003 conn(0x7fb5740fb900 msgr2=0x7fb5740fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 -- 172.21.15.23:0/4180575003 shutdown_connections 2023-11-16T23:55:25.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.440+0000 7fb579a21700 1 -- 172.21.15.23:0/4180575003 wait complete. 2023-11-16T23:55:25.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 Processor -- start 2023-11-16T23:55:25.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 -- start start 2023-11-16T23:55:25.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 0x7fb5741960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:25.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57419dc80 con 0x7fb574101e80 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57419ddc0 con 0x7fb5741014f0 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.441+0000 7fb579a21700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57419df50 con 0x7fb5741002f0 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40722/0 (socket says 172.21.15.23:40722) 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 -- 172.21.15.23:0/2668671225 learned_addr learned my addr 172.21.15.23:0/2668671225 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5727fc700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb572ffd700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 0x7fb5741960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 -- 172.21.15.23:0/2668671225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 msgr2=0x7fb5741960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 0x7fb5741960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 -- 172.21.15.23:0/2668671225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 msgr2=0x7fb574196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 -- 172.21.15.23:0/2668671225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c009d00 con 0x7fb574101e80 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5727fc700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.442+0000 7fb5737fe700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb56c007a60 tx=0x7fb56c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.443+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00e040 con 0x7fb574101e80 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.443+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb56c0094a0 con 0x7fb574101e80 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.443+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57410bc50 con 0x7fb574101e80 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.443+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0053b0 con 0x7fb574101e80 2023-11-16T23:55:25.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.443+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57410bac0 con 0x7fb574101e80 2023-11-16T23:55:25.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.444+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb574107860 con 0x7fb574101e80 2023-11-16T23:55:25.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.446+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb56c01d9c0 con 0x7fb574101e80 2023-11-16T23:55:25.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.447+0000 7fb578a1f700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 0x7fb55c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:25.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.448+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fb56c0cd0f0 con 0x7fb574101e80 2023-11-16T23:55:25.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.448+0000 7fb572ffd700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 0x7fb55c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:25.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.448+0000 7fb572ffd700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 0x7fb55c0a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb574101350 tx=0x7fb564007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.452+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb56c0976d0 con 0x7fb574101e80 2023-11-16T23:55:25.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:25.672+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb574061b00 con 0x7fb55c0a1150 2023-11-16T23:55:26.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:25 smithi023 ceph-mon[97824]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T23:55:26.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:25 smithi023 ceph-mon[97824]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:26.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:25 smithi173 ceph-mon[131534]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T23:55:26.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:25 smithi173 ceph-mon[131534]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:26.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:25 smithi156 ceph-mon[131105]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T23:55:26.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:25 smithi156 ceph-mon[131105]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:26.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:26 smithi023 ceph-mon[97824]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T23:55:26.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:26 smithi023 ceph-mon[97824]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:26.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:26.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:26.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:27.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:26 smithi173 ceph-mon[131534]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T23:55:27.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:26 smithi173 ceph-mon[131534]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:27.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:27.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:27.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:27.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:26 smithi156 ceph-mon[131105]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T23:55:27.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:26 smithi156 ceph-mon[131105]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:27.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:27.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:27.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:27.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:27 smithi023 ceph-mon[97824]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T23:55:27.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:27 smithi023 ceph-mon[97824]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:55:28.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:27 smithi173 ceph-mon[131534]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T23:55:28.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:27 smithi173 ceph-mon[131534]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:55:28.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:27 smithi156 ceph-mon[131105]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T23:55:28.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:27 smithi156 ceph-mon[131105]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:55:30.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:29 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]: dispatch 2023-11-16T23:55:30.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:29 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]': finished 2023-11-16T23:55:30.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:29 smithi023 ceph-mon[97824]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T23:55:30.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:30.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:29 smithi023 ceph-mon[97824]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:55:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:29 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]: dispatch 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:29 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]': finished 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:29 smithi173 ceph-mon[131534]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:29 smithi173 ceph-mon[131534]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:29 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]: dispatch 2023-11-16T23:55:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:29 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1209822722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b75c4c-cc8a-4327-bfda-9da89338654b"}]': finished 2023-11-16T23:55:30.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:29 smithi156 ceph-mon[131105]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T23:55:30.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:30.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:29 smithi156 ceph-mon[131105]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:55:31.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:30 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2942087284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:31.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:30 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2942087284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:30 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2942087284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:32.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:31 smithi023 ceph-mon[97824]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T23:55:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:31 smithi173 ceph-mon[131534]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T23:55:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:31 smithi156 ceph-mon[131105]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T23:55:34.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:34 smithi173 ceph-mon[131534]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T23:55:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:34 smithi156 ceph-mon[131105]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T23:55:34.501 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:34 smithi023 ceph-mon[97824]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-11-16T23:55:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:36 smithi173 ceph-mon[131534]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:36 smithi156 ceph-mon[131105]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:36.488 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:36 smithi023 ceph-mon[97824]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:37.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T23:55:37.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:37.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T23:55:37.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:37.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T23:55:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:38.119 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:38 smithi023 ceph-mon[97824]: Deploying daemon osd.6 on smithi023 2023-11-16T23:55:38.120 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:38 smithi023 ceph-mon[97824]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:38.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:38 smithi173 ceph-mon[131534]: Deploying daemon osd.6 on smithi023 2023-11-16T23:55:38.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:38 smithi173 ceph-mon[131534]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:38.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:38 smithi156 ceph-mon[131105]: Deploying daemon osd.6 on smithi023 2023-11-16T23:55:38.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:38 smithi156 ceph-mon[131105]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:39 smithi023 ceph-mon[97824]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:40.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:39 smithi173 ceph-mon[131534]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:39 smithi156 ceph-mon[131105]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:42.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:42 smithi173 ceph-mon[131534]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:42.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:42 smithi156 ceph-mon[131105]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:42 smithi023 ceph-mon[97824]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:44.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:44 smithi023 ceph-mon[97824]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:44.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.303 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:44 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:44 smithi156 ceph-mon[131105]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:44 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:44 smithi173 ceph-mon[131534]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:44.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:44 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:46.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:46 smithi023 ceph-mon[97824]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:46 smithi173 ceph-mon[131534]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:46.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:46 smithi156 ceph-mon[131105]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:46.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.730+0000 7fb578a1f700 1 -- 172.21.15.23:0/2668671225 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb574061b00 con 0x7fb55c0a1150 2023-11-16T23:55:46.731 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 msgr2=0x7fb55c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 0x7fb55c0a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb574101350 tx=0x7fb564007400 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 msgr2=0x7fb57419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb56c007a60 tx=0x7fb56c00bd60 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 shutdown_connections 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb574101e80 0x7fb57419d6b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb55c0a1150 0x7fb55c0a3600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5741002f0 0x7fb5741960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 --2- 172.21.15.23:0/2668671225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5741014f0 0x7fb574196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:46.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.733+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 >> 172.21.15.23:0/2668671225 conn(0x7fb5740fb900 msgr2=0x7fb5741049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:55:46.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.734+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 shutdown_connections 2023-11-16T23:55:46.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:55:46.734+0000 7fb579a21700 1 -- 172.21.15.23:0/2668671225 wait complete. 2023-11-16T23:55:47.139 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.6.service 2023-11-16T23:55:47.144 INFO:tasks.cephadm:Deploying osd.7 on smithi156 with /dev/vg_nvme/lv_2... 2023-11-16T23:55:47.144 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T23:55:47.224 INFO:journalctl@ceph.osd.6.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:47 smithi023 ceph-mon[97824]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:47 smithi173 ceph-mon[131534]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:47.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:55:47.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:47 smithi156 ceph-mon[131105]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T23:55:47.794 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-mon[97824]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.321 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:48.321 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T23:55:48.322 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T23:55:48.322 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:48.322 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.322 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:48 smithi156 ceph-mon[131105]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T23:55:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T23:55:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: from='osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:48 smithi173 ceph-mon[131534]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:55:48.770 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-16T23:55:48.723+0000 7fdfbd8ee700 -1 osd.6 0 waiting for initial osdmap 2023-11-16T23:55:48.771 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 16 23:55:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-16T23:55:48.735+0000 7fdfb8ee2700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00819664 s, 1.3 GB/s 2023-11-16T23:55:49.295 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-11-16T23:55:49.751 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2023-11-16T23:55:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:49 smithi023 ceph-mon[97824]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:55:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:49 smithi023 ceph-mon[97824]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T23:55:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:49 smithi023 ceph-mon[97824]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:49 smithi173 ceph-mon[131534]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:49 smithi173 ceph-mon[131534]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:49 smithi173 ceph-mon[131534]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:49 smithi156 ceph-mon[131105]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:49 smithi156 ceph-mon[131105]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T23:55:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:49 smithi156 ceph-mon[131105]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-11-16T23:55:50.388 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:55:50.979 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:55:50.979 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:55:50.979 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] boot 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi023 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:50.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi023 to 5757M 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:55:50.981 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:50 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:55:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:55:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:55:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] boot 2023-11-16T23:55:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T23:55:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi023 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi023 to 5757M 2023-11-16T23:55:51.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:51.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:51.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:51.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:55:51.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:50 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:55:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: osd.6 [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] boot 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi023 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi023 to 5757M 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:55:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:55:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:50 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:55:51.686 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 -- 172.21.15.156:0/3118531210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc100d70 msgr2=0x7fbecc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/3118531210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc100d70 0x7fbecc107630 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbec400b0a0 tx=0x7fbec400b3b0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 -- 172.21.15.156:0/3118531210 shutdown_connections 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/3118531210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc100d70 0x7fbecc107630 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/3118531210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 0x7fbecc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/3118531210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc1003e0 0x7fbecc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 -- 172.21.15.156:0/3118531210 >> 172.21.15.156:0/3118531210 conn(0x7fbecc0fa7f0 msgr2=0x7fbecc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 -- 172.21.15.156:0/3118531210 shutdown_connections 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.682+0000 7fbed2c0b700 1 -- 172.21.15.156:0/3118531210 wait complete. 2023-11-16T23:55:51.687 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 Processor -- start 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 -- start start 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 0x7fbecc069a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 0x7fbecc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc19dc20 con 0x7fbecc100d70 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc19dd90 con 0x7fbecc1003e0 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed2c0b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc19df50 con 0x7fbecc0ff1e0 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35378/0 (socket says 172.21.15.156:35378) 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 -- 172.21.15.156:0/2424170414 learned_addr learned my addr 172.21.15.156:0/2424170414 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed09a7700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 0x7fbecc069a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:51.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbed11a8700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 0x7fbecc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 -- 172.21.15.156:0/2424170414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 msgr2=0x7fbecc069a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 0x7fbecc069a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 -- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 msgr2=0x7fbecc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.683+0000 7fbecbfff700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 0x7fbecc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbecbfff700 1 -- 172.21.15.156:0/2424170414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec4009d00 con 0x7fbecc1003e0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbed11a8700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 0x7fbecc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbecbfff700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800c920 tx=0x7fbeb800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8007b00 con 0x7fbecc1003e0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeb800ce70 con 0x7fbecc1003e0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc10ab30 con 0x7fbecc1003e0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb80054e0 con 0x7fbecc1003e0 2023-11-16T23:55:51.689 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.684+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc10a9a0 con 0x7fbecc1003e0 2023-11-16T23:55:51.690 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.685+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc106740 con 0x7fbecc1003e0 2023-11-16T23:55:51.690 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.686+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbeb8004750 con 0x7fbecc1003e0 2023-11-16T23:55:51.691 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.686+0000 7fbec9ffb700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 0x7fbebc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:55:51.691 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.687+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fbeb80cbb50 con 0x7fbecc1003e0 2023-11-16T23:55:51.691 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.687+0000 7fbed09a7700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 0x7fbebc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:55:51.691 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.687+0000 7fbed09a7700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 0x7fbebc0a3680 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbecc100240 tx=0x7fbec000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:55:51.691 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.690+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbeb8096030 con 0x7fbecc1003e0 2023-11-16T23:55:51.886 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:55:51.885+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fbecc061c80 con 0x7fbebc0a11d0 2023-11-16T23:55:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:51 smithi023 ceph-mon[97824]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T23:55:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:51 smithi023 ceph-mon[97824]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:52.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:51 smithi173 ceph-mon[131534]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T23:55:52.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:51 smithi173 ceph-mon[131534]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:51 smithi156 ceph-mon[131105]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T23:55:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:51 smithi156 ceph-mon[131105]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:52 smithi023 ceph-mon[97824]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:53.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:52 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:53.026 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:52 smithi156 ceph-mon[131105]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:53.026 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:53.026 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:53.026 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:52 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:53.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:52 smithi173 ceph-mon[131534]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:55:53.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:55:53.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:55:53.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:52 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:55:54.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:53 smithi023 ceph-mon[97824]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:54.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:53 smithi156 ceph-mon[131105]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:54.112 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:53 smithi173 ceph-mon[131534]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:55.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:54 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/2691058871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:54 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:54 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]': finished 2023-11-16T23:55:55.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:54 smithi023 ceph-mon[97824]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T23:55:55.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:55:55.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:54 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/2691058871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:54 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:54 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]': finished 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:54 smithi173 ceph-mon[131534]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:54 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/2691058871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:54 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]: dispatch 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:54 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f9a70dc-a716-42aa-a798-635dd1f18ce1"}]': finished 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:54 smithi156 ceph-mon[131105]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T23:55:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:55:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:55 smithi023 ceph-mon[97824]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:55 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/1892828623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:56.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:55 smithi173 ceph-mon[131534]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:56.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:55 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/1892828623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:56.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:55 smithi156 ceph-mon[131105]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:56.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:55 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/1892828623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:55:58.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:58 smithi173 ceph-mon[131534]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:58.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:58 smithi156 ceph-mon[131105]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:58.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:58 smithi023 ceph-mon[97824]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:55:59.844 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:55:59 smithi156 ceph-mon[131105]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:55:59 smithi023 ceph-mon[97824]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:00.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:55:59 smithi173 ceph-mon[131534]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:02.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:02 smithi173 ceph-mon[131534]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:02.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T23:56:02.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:02.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:02 smithi156 ceph-mon[131105]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:02.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T23:56:02.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:02.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:02 smithi023 ceph-mon[97824]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T23:56:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:03.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:03 smithi156 ceph-mon[131105]: Deploying daemon osd.7 on smithi156 2023-11-16T23:56:03.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:03 smithi173 ceph-mon[131534]: Deploying daemon osd.7 on smithi156 2023-11-16T23:56:03.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:03 smithi023 ceph-mon[97824]: Deploying daemon osd.7 on smithi156 2023-11-16T23:56:04.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:04 smithi173 ceph-mon[131534]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:04.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:04 smithi156 ceph-mon[131105]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:04.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:04 smithi023 ceph-mon[97824]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:06.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:06 smithi173 ceph-mon[131534]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:06 smithi156 ceph-mon[131105]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:06.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:06 smithi023 ceph-mon[97824]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:06.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:08.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:08 smithi173 ceph-mon[131534]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:08.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:08 smithi156 ceph-mon[131105]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:08.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:08 smithi023 ceph-mon[97824]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:08.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.602+0000 7fbec9ffb700 1 -- 172.21.15.156:0/2424170414 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbecc061c80 con 0x7fbebc0a11d0 2023-11-16T23:56:08.604 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 7 on host 'smithi156' 2023-11-16T23:56:08.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 msgr2=0x7fbebc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:08.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 0x7fbebc0a3680 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbecc100240 tx=0x7fbec000b410 comp rx=0 tx=0).stop 2023-11-16T23:56:08.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 msgr2=0x7fbecc19adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800c920 tx=0x7fbeb800cc30 comp rx=0 tx=0).stop 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 shutdown_connections 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbecc100d70 0x7fbecc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbebc0a11d0 0x7fbebc0a3680 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbecc0ff1e0 0x7fbecc069a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 --2- 172.21.15.156:0/2424170414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbecc1003e0 0x7fbecc19adf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 >> 172.21.15.156:0/2424170414 conn(0x7fbecc0fa7f0 msgr2=0x7fbecc103880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 shutdown_connections 2023-11-16T23:56:08.606 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:56:08.604+0000 7fbed2c0b700 1 -- 172.21.15.156:0/2424170414 wait complete. 2023-11-16T23:56:09.088 DEBUG:teuthology.orchestra.run.smithi156:osd.7> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.7.service 2023-11-16T23:56:09.090 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_2... 2023-11-16T23:56:09.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T23:56:09.123 INFO:journalctl@ceph.osd.7.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:56:09.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:09.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:09 smithi173 ceph-mon[131534]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.637 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:09.638 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-mon[131105]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:09.740 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:09.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:09 smithi023 ceph-mon[97824]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:09.901 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 16 23:56:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-16T23:56:09.785+0000 7f78d62f2640 -1 osd.7 0 log_to_monitors true 2023-11-16T23:56:10.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:10 smithi173 ceph-mon[131534]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:10.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:10 smithi173 ceph-mon[131534]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:10.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:10 smithi156 ceph-mon[131105]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:10.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:10 smithi156 ceph-mon[131105]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:11.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:10 smithi023 ceph-mon[97824]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:11.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:10 smithi023 ceph-mon[97824]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T23:56:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T23:56:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T23:56:11.339 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2023-11-16T23:56:11.339 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2023-11-16T23:56:11.339 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012466 s, 841 MB/s 2023-11-16T23:56:11.339 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2023-11-16T23:56:11.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2023-11-16T23:56:12.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T23:56:12.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T23:56:12.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.108 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-16T23:56:11.661+0000 7f78cb969700 -1 osd.7 0 waiting for initial osdmap 2023-11-16T23:56:12.108 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-16T23:56:11.672+0000 7f78c3f57700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:56:12.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T23:56:12.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T23:56:12.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:12.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:12.421 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi156 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi156 to 5722M 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] boot 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T23:56:13.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi156 2023-11-16T23:56:13.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi156 to 5722M 2023-11-16T23:56:13.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:56:13.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:56:13.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T23:56:13.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] boot 2023-11-16T23:56:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T23:56:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi156 2023-11-16T23:56:13.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi156 to 5722M 2023-11-16T23:56:13.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: osd.7 [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] boot 2023-11-16T23:56:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T23:56:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T23:56:13.766 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 -- 172.21.15.173:0/1266807859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ff500 msgr2=0x7fe7a40ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1266807859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ff500 0x7fe7a40ff950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7980099c0 tx=0x7fe798009cd0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 -- 172.21.15.173:0/1266807859 shutdown_connections 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1266807859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a40ffe90 0x7fe7a4107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1266807859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a41007c0 0x7fe7a4100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1266807859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ff500 0x7fe7a40ff950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 -- 172.21.15.173:0/1266807859 >> 172.21.15.173:0/1266807859 conn(0x7fe7a40fa000 msgr2=0x7fe7a40fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 -- 172.21.15.173:0/1266807859 shutdown_connections 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.765+0000 7fe7a931f700 1 -- 172.21.15.173:0/1266807859 wait complete. 2023-11-16T23:56:13.767 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.766+0000 7fe7a931f700 1 Processor -- start 2023-11-16T23:56:13.769 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.766+0000 7fe7a931f700 1 -- start start 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a40ff500 0x7fe7a40fef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 0x7fe7a419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a419db90 con 0x7fe7a41007c0 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a419dd00 con 0x7fe7a40ff500 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a419dec0 con 0x7fe7a40ffe90 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51318/0 (socket says 172.21.15.173:51318) 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 -- 172.21.15.173:0/1938838443 learned_addr learned my addr 172.21.15.173:0/1938838443 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a2ffd700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a40ff500 0x7fe7a40fef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a37fe700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 0x7fe7a419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:13.770 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 -- 172.21.15.173:0/1938838443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a40ff500 msgr2=0x7fe7a40fef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a40ff500 0x7fe7a40fef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 -- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 msgr2=0x7fe7a419d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 0x7fe7a419d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 -- 172.21.15.173:0/1938838443 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980095f0 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a37fe700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 0x7fe7a419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a27fc700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe79800c010 tx=0x7fe798004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79801d070 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a410abf0 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe798022470 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.767+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79800f6c0 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.768+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a410aa60 con 0x7fe7a40ffe90 2023-11-16T23:56:13.771 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.769+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a4106890 con 0x7fe7a40ffe90 2023-11-16T23:56:13.772 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.770+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe7980225e0 con 0x7fe7a40ffe90 2023-11-16T23:56:13.772 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.770+0000 7fe78bfff700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 0x7fe78c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:13.772 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.770+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fe7980cda70 con 0x7fe7a40ffe90 2023-11-16T23:56:13.772 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.770+0000 7fe7a2ffd700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 0x7fe78c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:13.773 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.771+0000 7fe7a2ffd700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 0x7fe78c0a3600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4068e40 tx=0x7fe79400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:56:13.774 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.774+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe798097d90 con 0x7fe7a40ffe90 2023-11-16T23:56:13.964 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:13.963+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7a4061bf0 con 0x7fe78c0a1150 2023-11-16T23:56:14.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:13 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:56:14.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:13 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:56:14.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:13 smithi023 ceph-mon[97824]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:14.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:13 smithi023 ceph-mon[97824]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T23:56:14.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:13 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:56:14.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:13 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:13 smithi173 ceph-mon[131534]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:13 smithi173 ceph-mon[131534]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:13 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:13 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:13 smithi156 ceph-mon[131105]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:14.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:13 smithi156 ceph-mon[131105]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T23:56:14.932 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:14 smithi173 ceph-mon[131534]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:14.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:14.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:14.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:14.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:14 smithi173 ceph-mon[131534]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T23:56:15.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:14 smithi023 ceph-mon[97824]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:15.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:15.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:15.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:15.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:14 smithi023 ceph-mon[97824]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T23:56:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:14 smithi156 ceph-mon[131105]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:14 smithi156 ceph-mon[131105]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T23:56:15.770 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:15 smithi173 ceph-mon[131534]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:16.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:15 smithi023 ceph-mon[97824]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:16.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:15 smithi156 ceph-mon[131105]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:17.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T23:56:17.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/3254006745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]': finished 2023-11-16T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:17.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/3254006745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]': finished 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/3254006745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]: dispatch 2023-11-16T23:56:17.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31074fdd-3ccc-460e-91e1-10339bf34c95"}]': finished 2023-11-16T23:56:17.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: osdmap e55: 9 total, 8 up, 9 in 2023-11-16T23:56:17.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:18.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:17 smithi023 ceph-mon[97824]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:18.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:17 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/4100157226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:18.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:17 smithi173 ceph-mon[131534]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:18.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:17 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/4100157226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:18.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:17 smithi156 ceph-mon[131105]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:18.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:17 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/4100157226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:19 smithi023 ceph-mon[97824]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:19 smithi173 ceph-mon[131534]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:19 smithi156 ceph-mon[131105]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:22 smithi173 ceph-mon[131534]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:22.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:22 smithi156 ceph-mon[131105]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:22.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:22 smithi023 ceph-mon[97824]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:23.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T23:56:23.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:23.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T23:56:23.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:23.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T23:56:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:24.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:24 smithi173 ceph-mon[131534]: Deploying daemon osd.8 on smithi173 2023-11-16T23:56:24.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:24 smithi173 ceph-mon[131534]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:24 smithi156 ceph-mon[131105]: Deploying daemon osd.8 on smithi173 2023-11-16T23:56:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:24 smithi156 ceph-mon[131105]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:24.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:24 smithi023 ceph-mon[97824]: Deploying daemon osd.8 on smithi173 2023-11-16T23:56:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:24 smithi023 ceph-mon[97824]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:26.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:26 smithi173 ceph-mon[131534]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:26.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:26 smithi156 ceph-mon[131105]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:26.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:26 smithi023 ceph-mon[97824]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:28.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:28 smithi173 ceph-mon[131534]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:28.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:28.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:28 smithi156 ceph-mon[131105]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:28.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:28.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:28 smithi023 ceph-mon[97824]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:29.834 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:29 smithi173 ceph-mon[131534]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:30.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:29 smithi023 ceph-mon[97824]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:29 smithi156 ceph-mon[131105]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:30.831 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.829+0000 7fe78bfff700 1 -- 172.21.15.173:0/1938838443 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe7a4061bf0 con 0x7fe78c0a1150 2023-11-16T23:56:30.831 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2023-11-16T23:56:30.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.831+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 msgr2=0x7fe78c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:30.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 0x7fe78c0a3600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4068e40 tx=0x7fe79400a380 comp rx=0 tx=0).stop 2023-11-16T23:56:30.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 msgr2=0x7fe7a419ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:30.833 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe79800c010 tx=0x7fe798004730 comp rx=0 tx=0).stop 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 shutdown_connections 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7a41007c0 0x7fe7a419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a40ff500 0x7fe7a40fef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe78c0a1150 0x7fe78c0a3600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 --2- 172.21.15.173:0/1938838443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a40ffe90 0x7fe7a419ada0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 >> 172.21.15.173:0/1938838443 conn(0x7fe7a40fa000 msgr2=0x7fe7a41039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 shutdown_connections 2023-11-16T23:56:30.834 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:56:30.832+0000 7fe7a931f700 1 -- 172.21.15.173:0/1938838443 wait complete. 2023-11-16T23:56:31.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:31.316 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.8.service 2023-11-16T23:56:31.322 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2023-11-16T23:56:31.322 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T23:56:31.350 INFO:journalctl@ceph.osd.8.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:31.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:31.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:31.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:31.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:56:31.838 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 16 23:56:31 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-16T23:56:31.575+0000 7fd2d3eac640 -1 osd.8 0 log_to_monitors true 2023-11-16T23:56:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:32 smithi173 ceph-mon[131534]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:32 smithi173 ceph-mon[131534]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:32 smithi173 ceph-mon[131534]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:32.196 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:56:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:32 smithi156 ceph-mon[131105]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:32 smithi156 ceph-mon[131105]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:32 smithi156 ceph-mon[131105]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:32.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:32 smithi023 ceph-mon[97824]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:32.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:32 smithi023 ceph-mon[97824]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:32.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:32 smithi023 ceph-mon[97824]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T23:56:33.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-mon[131534]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T23:56:33.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-mon[131534]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T23:56:33.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:33.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-mon[131534]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:33.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-mon[131534]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:33 smithi023 ceph-mon[97824]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T23:56:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:33 smithi023 ceph-mon[97824]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T23:56:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:33 smithi023 ceph-mon[97824]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:33 smithi023 ceph-mon[97824]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:33.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-16T23:56:33.076+0000 7fd2c9523700 -1 osd.8 0 waiting for initial osdmap 2023-11-16T23:56:33.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 16 23:56:33 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-16T23:56:33.089+0000 7fd2c2312700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:56:33.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:33 smithi156 ceph-mon[131105]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T23:56:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:33 smithi156 ceph-mon[131105]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T23:56:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:33 smithi156 ceph-mon[131105]: from='osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:33 smithi156 ceph-mon[131105]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:56:34.273 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T23:56:34.274 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T23:56:34.274 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2023-11-16T23:56:34.274 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2023-11-16T23:56:34.274 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0308741 s, 340 MB/s 2023-11-16T23:56:34.274 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2023-11-16T23:56:34.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] boot 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:56:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] boot 2023-11-16T23:56:34.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T23:56:34.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.413 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.414 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:34.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:56:34.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:56:34.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: osd.8 [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] boot 2023-11-16T23:56:34.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: osdmap e58: 9 total, 9 up, 9 in 2023-11-16T23:56:34.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T23:56:34.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi173 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi173 to 5722M 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:35 smithi156 ceph-mon[131105]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:56:35.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi173 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi173 to 5722M 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:35 smithi173 ceph-mon[131534]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T23:56:35.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi173 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi173 to 5722M 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:56:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:35 smithi023 ceph-mon[97824]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T23:56:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:56:36.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:36 smithi023 ceph-mon[97824]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:36.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:36 smithi023 ceph-mon[97824]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T23:56:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:36 smithi173 ceph-mon[131534]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:36 smithi173 ceph-mon[131534]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T23:56:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:36 smithi156 ceph-mon[131105]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:36 smithi156 ceph-mon[131105]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T23:56:37.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.144+0000 7efdb1899700 1 -- 172.21.15.23:0/234008881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac1081c0 msgr2=0x7efdac10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.144+0000 7efdb1899700 1 --2- 172.21.15.23:0/234008881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac1081c0 0x7efdac10a600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efda400b0a0 tx=0x7efda400b3b0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 -- 172.21.15.23:0/234008881 shutdown_connections 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 --2- 172.21.15.23:0/234008881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac1081c0 0x7efdac10a600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 --2- 172.21.15.23:0/234008881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac074b20 0x7efdac072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 --2- 172.21.15.23:0/234008881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 0x7efdac0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 -- 172.21.15.23:0/234008881 >> 172.21.15.23:0/234008881 conn(0x7efdac0fb900 msgr2=0x7efdac0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 -- 172.21.15.23:0/234008881 shutdown_connections 2023-11-16T23:56:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.145+0000 7efdb1899700 1 -- 172.21.15.23:0/234008881 wait complete. 2023-11-16T23:56:37.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 Processor -- start 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 -- start start 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 0x7efdac198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac1081c0 0x7efdac19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac19fec0 con 0x7efdac074b20 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac1a0000 con 0x7efdac073550 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.146+0000 7efdb1899700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac1a0190 con 0x7efdac1081c0 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60698/0 (socket says 172.21.15.23:60698) 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 -- 172.21.15.23:0/3035024792 learned_addr learned my addr 172.21.15.23:0/3035024792 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdab7fe700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac1081c0 0x7efdac19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaaffd700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 0x7efdac198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 -- 172.21.15.23:0/3035024792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac1081c0 msgr2=0x7efdac19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac1081c0 0x7efdac19f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 -- 172.21.15.23:0/3035024792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 msgr2=0x7efdac198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 0x7efdac198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 -- 172.21.15.23:0/3035024792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4009d00 con 0x7efdac074b20 2023-11-16T23:56:37.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.147+0000 7efdaa7fc700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9d0 tx=0x7efda000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.148+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0009790 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.148+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efda000ee70 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.148+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdac10deb0 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.148+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0010ea0 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.148+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdac10dd20 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.149+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdac106f80 con 0x7efdac074b20 2023-11-16T23:56:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.151+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efda000cbd0 con 0x7efdac074b20 2023-11-16T23:56:37.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.152+0000 7efdb0897700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 0x7efd940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:56:37.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.152+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5829+0+0 (secure 0 0 0) 0x7efda0014070 con 0x7efdac074b20 2023-11-16T23:56:37.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.153+0000 7efdaaffd700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 0x7efd940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:56:37.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.153+0000 7efdaaffd700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 0x7efd940a3600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd9c004630 tx=0x7efd9c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:56:37.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.157+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efda0051730 con 0x7efdac074b20 2023-11-16T23:56:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:37.377+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efdac061a30 con 0x7efd940a1150 2023-11-16T23:56:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:37 smithi023 ceph-mon[97824]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T23:56:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:37 smithi023 ceph-mon[97824]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:37 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:38.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:37 smithi173 ceph-mon[131534]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T23:56:38.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:37 smithi173 ceph-mon[131534]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:38.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:38.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:38.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:37 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:38.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:37 smithi156 ceph-mon[131105]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T23:56:38.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:37 smithi156 ceph-mon[131105]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:38.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:56:38.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:56:38.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:37 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:38.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:38 smithi023 ceph-mon[97824]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:38.879 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:38 smithi023 ceph-mon[97824]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T23:56:39.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:38 smithi173 ceph-mon[131534]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:39.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:38 smithi173 ceph-mon[131534]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T23:56:39.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:38 smithi156 ceph-mon[131105]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:56:39.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:38 smithi156 ceph-mon[131105]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T23:56:39.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:39 smithi023 ceph-mon[97824]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:56:40.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:39 smithi173 ceph-mon[131534]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:56:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:39 smithi156 ceph-mon[131105]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T23:56:41.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]: dispatch 2023-11-16T23:56:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]': finished 2023-11-16T23:56:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:40 smithi023 ceph-mon[97824]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T23:56:41.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:40 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:56:41.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:40 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]: dispatch 2023-11-16T23:56:41.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:40 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]': finished 2023-11-16T23:56:41.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:40 smithi173 ceph-mon[131534]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T23:56:41.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:40 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:56:41.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:40 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]: dispatch 2023-11-16T23:56:41.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:40 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2469280607' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87031132-5932-4aa2-ad9d-4b706edfac8f"}]': finished 2023-11-16T23:56:41.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:40 smithi156 ceph-mon[131105]: osdmap e63: 10 total, 9 up, 10 in 2023-11-16T23:56:41.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:40 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:56:42.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:41 smithi023 ceph-mon[97824]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:56:42.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:41 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1259576336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:42.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:41 smithi173 ceph-mon[131534]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:56:42.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:41 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1259576336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:42.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:41 smithi156 ceph-mon[131105]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-11-16T23:56:42.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:41 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1259576336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:56:44.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:44 smithi173 ceph-mon[131534]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:56:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:44 smithi156 ceph-mon[131105]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:56:44.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:44 smithi023 ceph-mon[97824]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:56:46.336 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:46 smithi023 ceph-mon[97824]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:56:46.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:46 smithi173 ceph-mon[131534]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:56:46.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:46 smithi156 ceph-mon[131105]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:56:48.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:48 smithi023 ceph-mon[97824]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T23:56:48.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T23:56:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:48 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:48 smithi156 ceph-mon[131105]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:48 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:48 smithi173 ceph-mon[131534]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T23:56:48.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:48 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:56:49.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:49 smithi023 ceph-mon[97824]: Deploying daemon osd.9 on smithi023 2023-11-16T23:56:49.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:49 smithi173 ceph-mon[131534]: Deploying daemon osd.9 on smithi023 2023-11-16T23:56:49.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:49 smithi156 ceph-mon[131105]: Deploying daemon osd.9 on smithi023 2023-11-16T23:56:50.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:50 smithi023 ceph-mon[97824]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:50.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:50 smithi173 ceph-mon[131534]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:50.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:50 smithi156 ceph-mon[131105]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:52.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:52 smithi173 ceph-mon[131534]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:52.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:52 smithi156 ceph-mon[131105]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:52 smithi023 ceph-mon[97824]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:54.246 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:54 smithi023 ceph-mon[97824]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:56:54.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:54 smithi173 ceph-mon[131534]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:55 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.122 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:55 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:55 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:56 smithi023 ceph-mon[97824]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.936+0000 7efdb0897700 1 -- 172.21.15.23:0/3035024792 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efdac061a30 con 0x7efd940a1150 2023-11-16T23:57:08.124 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2023-11-16T23:57:08.125 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:55 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.200 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:55 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:55 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:56 smithi173 ceph-mon[131534]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.519 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:58 smithi173 ceph-mon[131534]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.522 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:56:59 smithi173 ceph-mon[131534]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.522 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:00 smithi173 ceph-mon[131534]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:00 smithi173 ceph-mon[131534]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:00 smithi173 ceph-mon[131534]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:00 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:57:08.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] boot 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:01 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi023 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.590 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.614 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.642 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.671 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.671 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi023 to 4317M 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:02 smithi173 ceph-mon[131534]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:03 smithi173 ceph-mon[131534]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:08.696 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:06 smithi173 ceph-mon[131534]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:54 smithi156 ceph-mon[131105]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:55 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:55 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:55 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:56 smithi156 ceph-mon[131105]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:58 smithi156 ceph-mon[131105]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T23:57:08.698 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:56:59 smithi156 ceph-mon[131105]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:00 smithi156 ceph-mon[131105]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:00 smithi156 ceph-mon[131105]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:57:08.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:00 smithi156 ceph-mon[131105]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:00 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] boot 2023-11-16T23:57:08.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:01 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi023 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi023 to 4317M 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.842 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:02 smithi156 ceph-mon[131105]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:03 smithi156 ceph-mon[131105]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:08.843 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:06 smithi156 ceph-mon[131105]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 msgr2=0x7efd940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 0x7efd940a3600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd9c004630 tx=0x7efd9c00b3c0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 msgr2=0x7efdac198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9d0 tx=0x7efda000ece0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 shutdown_connections 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7efd940a1150 0x7efd940a3600 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdac1081c0 0x7efdac19f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdac073550 0x7efdac198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 --2- 172.21.15.23:0/3035024792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac074b20 0x7efdac198870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 >> 172.21.15.23:0/3035024792 conn(0x7efdac0fb900 msgr2=0x7efdac1040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 shutdown_connections 2023-11-16T23:57:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:56:57.939+0000 7efdb1899700 1 -- 172.21.15.23:0/3035024792 wait complete. 2023-11-16T23:57:08.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.874 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.875 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.907 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.907 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:08.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:58 smithi023 ceph-mon[97824]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T23:57:08.910 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.9.service 2023-11-16T23:57:08.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T23:57:08.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T23:57:08.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: from='osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:56:59 smithi023 ceph-mon[97824]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:00 smithi023 ceph-mon[97824]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:00 smithi023 ceph-mon[97824]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:00 smithi023 ceph-mon[97824]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T23:57:08.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:00 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:57:08.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:57:08.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: osd.9 [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] boot 2023-11-16T23:57:08.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: osdmap e66: 10 total, 10 up, 10 in 2023-11-16T23:57:08.939 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:01 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T23:57:08.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-11-16T23:57:08.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi023 2023-11-16T23:57:08.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.965 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi023 to 4317M 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:02 smithi023 ceph-mon[97824]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T23:57:08.968 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:03 smithi023 ceph-mon[97824]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:08.968 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:06 smithi023 ceph-mon[97824]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:09.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:08 smithi023 ceph-mon[97824]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:09.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:08 smithi156 ceph-mon[131105]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:09.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:08 smithi173 ceph-mon[131534]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:09.014 INFO:tasks.cephadm:Deploying osd.10 on smithi156 with /dev/vg_nvme/lv_1... 2023-11-16T23:57:09.014 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T23:57:09.095 INFO:journalctl@ceph.osd.9.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:57:09.667 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:57:09.937 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:09 smithi156 ceph-mon[131105]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:10.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:09 smithi023 ceph-mon[97824]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:10.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:09 smithi173 ceph-mon[131534]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:11.156 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T23:57:11.156 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T23:57:11.156 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-11-16T23:57:11.156 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-11-16T23:57:11.156 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00806057 s, 1.3 GB/s 2023-11-16T23:57:11.157 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 2023-11-16T23:57:11.157 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-11-16T23:57:11.523 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2023-11-16T23:57:12.165 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T23:57:12.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:12 smithi173 ceph-mon[131534]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:12 smithi156 ceph-mon[131105]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:12 smithi023 ceph-mon[97824]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:13.370 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 -- 172.21.15.156:0/994759096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee81003e0 msgr2=0x7f2ee8100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:13.370 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/994759096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee81003e0 0x7f2ee8100830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4009990 tx=0x7f2ed4009ca0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.370 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 -- 172.21.15.156:0/994759096 shutdown_connections 2023-11-16T23:57:13.370 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/994759096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ee8100d70 0x7f2ee8107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.370 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/994759096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee80ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.371 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/994759096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee81003e0 0x7f2ee8100830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.371 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 -- 172.21.15.156:0/994759096 >> 172.21.15.156:0/994759096 conn(0x7f2ee80fa7f0 msgr2=0x7f2ee80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:13.371 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 -- 172.21.15.156:0/994759096 shutdown_connections 2023-11-16T23:57:13.371 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.369+0000 7f2eeec0c700 1 -- 172.21.15.156:0/994759096 wait complete. 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 Processor -- start 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 -- start start 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ee81003e0 0x7f2ee8195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 0x7f2ee819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee819ca70 con 0x7f2ee81003e0 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee819cbb0 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eeec0c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee819cd40 con 0x7f2ee8100d70 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:13.373 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35516/0 (socket says 172.21.15.156:35516) 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 -- 172.21.15.156:0/3042234438 learned_addr learned my addr 172.21.15.156:0/3042234438 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eed1a9700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 0x7f2ee819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 -- 172.21.15.156:0/3042234438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 msgr2=0x7f2ee819c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 0x7f2ee819c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 -- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ee81003e0 msgr2=0x7f2ee8195420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ee81003e0 0x7f2ee8195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eec9a8700 1 -- 172.21.15.156:0/3042234438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40095f0 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.370+0000 7f2eed1a9700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 0x7f2ee819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2eec9a8700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00c990 tx=0x7f2edc00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc007b00 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2edc007c60 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.374 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee810ac00 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.375 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc018660 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.375 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.371+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee810aa70 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.375 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.372+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee819d390 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.375 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.373+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2edc00f450 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.722 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.374+0000 7f2ee5ffb700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 0x7f2ed80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:13.722 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.374+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f2edc0cc160 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.722 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.374+0000 7f2ee7fff700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 0x7f2ed80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:13.722 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.374+0000 7f2ee7fff700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 0x7f2ed80a3680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4005f50 tx=0x7f2ed4005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:13.723 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.378+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2edc096010 con 0x7f2ee80ff1e0 2023-11-16T23:57:13.723 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:13.571+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ee819d940 con 0x7f2ed80a11d0 2023-11-16T23:57:14.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:14 smithi173 ceph-mon[131534]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:14.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:14 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:14 smithi156 ceph-mon[131105]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:14.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:14 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:14 smithi023 ceph-mon[97824]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:14 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:15.356 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:15 smithi156 ceph-mon[131105]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:15.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:15 smithi173 ceph-mon[131534]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:15.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:15 smithi023 ceph-mon[97824]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:16.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:16.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/3269020536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]': finished 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/3269020536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]': finished 2023-11-16T23:57:16.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T23:57:16.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:16.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/3269020536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]: dispatch 2023-11-16T23:57:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afa77aab-17f7-4ab5-a277-c10102ad7e5d"}]': finished 2023-11-16T23:57:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: osdmap e68: 11 total, 10 up, 11 in 2023-11-16T23:57:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:17.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:17 smithi023 ceph-mon[97824]: from='client.? 172.21.15.156:0/213716443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:17.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:17 smithi173 ceph-mon[131534]: from='client.? 172.21.15.156:0/213716443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:17.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:17 smithi156 ceph-mon[131105]: from='client.? 172.21.15.156:0/213716443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:18.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:18 smithi023 ceph-mon[97824]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:18.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:18 smithi173 ceph-mon[131534]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:18 smithi156 ceph-mon[131105]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:20.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:19 smithi023 ceph-mon[97824]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:19 smithi173 ceph-mon[131534]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:19 smithi156 ceph-mon[131105]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:22 smithi173 ceph-mon[131534]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:22.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:22 smithi156 ceph-mon[131105]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:22.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:22 smithi023 ceph-mon[97824]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:23.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T23:57:23.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:23.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T23:57:23.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T23:57:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:24.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:24 smithi173 ceph-mon[131534]: Deploying daemon osd.10 on smithi156 2023-11-16T23:57:24.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:24 smithi173 ceph-mon[131534]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:24 smithi156 ceph-mon[131105]: Deploying daemon osd.10 on smithi156 2023-11-16T23:57:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:24 smithi156 ceph-mon[131105]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:24.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:24 smithi023 ceph-mon[97824]: Deploying daemon osd.10 on smithi156 2023-11-16T23:57:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:24 smithi023 ceph-mon[97824]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:26.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:26 smithi173 ceph-mon[131534]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:26.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:26 smithi156 ceph-mon[131105]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:26.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:26 smithi023 ceph-mon[97824]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:28.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:28 smithi023 ceph-mon[97824]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:28 smithi173 ceph-mon[131534]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:28 smithi156 ceph-mon[131105]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:29.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:29 smithi156 ceph-mon[131105]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:30.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:29 smithi023 ceph-mon[97824]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:29 smithi173 ceph-mon[131534]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:30.435 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.435+0000 7f2ee5ffb700 1 -- 172.21.15.156:0/3042234438 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f2ee819d940 con 0x7f2ed80a11d0 2023-11-16T23:57:30.436 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 10 on host 'smithi156' 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 msgr2=0x7f2ed80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 0x7f2ed80a3680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4005f50 tx=0x7f2ed4005e80 comp rx=0 tx=0).stop 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 msgr2=0x7f2ee8194ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00c990 tx=0x7f2edc00cd50 comp rx=0 tx=0).stop 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 shutdown_connections 2023-11-16T23:57:30.438 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2ed80a11d0 0x7f2ed80a3680 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8100d70 0x7f2ee819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ee80ff1e0 0x7f2ee8194ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 --2- 172.21.15.156:0/3042234438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ee81003e0 0x7f2ee8195420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 >> 172.21.15.156:0/3042234438 conn(0x7f2ee80fa7f0 msgr2=0x7f2ee80fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 shutdown_connections 2023-11-16T23:57:30.439 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-16T23:57:30.438+0000 7f2eeec0c700 1 -- 172.21.15.156:0/3042234438 wait complete. 2023-11-16T23:57:30.863 DEBUG:teuthology.orchestra.run.smithi156:osd.10> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.10.service 2023-11-16T23:57:30.865 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2023-11-16T23:57:30.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T23:57:30.894 INFO:journalctl@ceph.osd.10.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:31.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:31.408 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 16 23:57:31 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-16T23:57:31.149+0000 7f27ca243640 -1 osd.10 0 log_to_monitors true 2023-11-16T23:57:31.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:57:32.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:32 smithi173 ceph-mon[131534]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:32.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:32 smithi173 ceph-mon[131534]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:32.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:32 smithi173 ceph-mon[131534]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:32 smithi156 ceph-mon[131105]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:32 smithi156 ceph-mon[131105]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:32.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:32 smithi156 ceph-mon[131105]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:32.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:32 smithi023 ceph-mon[97824]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:32 smithi023 ceph-mon[97824]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:32 smithi023 ceph-mon[97824]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0153524 s, 683 MB/s 2023-11-16T23:57:33.150 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2023-11-16T23:57:33.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:33 smithi173 ceph-mon[131534]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T23:57:33.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:33 smithi173 ceph-mon[131534]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T23:57:33.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:33.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:33 smithi173 ceph-mon[131534]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:33 smithi173 ceph-mon[131534]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-16T23:57:33.102+0000 7f27c00bb700 -1 osd.10 0 waiting for initial osdmap 2023-11-16T23:57:33.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-16T23:57:33.116+0000 7f27b76a7700 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:57:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-mon[131105]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T23:57:33.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-mon[131105]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T23:57:33.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:33.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-mon[131105]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:33 smithi156 ceph-mon[131105]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:33 smithi023 ceph-mon[97824]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T23:57:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:33 smithi023 ceph-mon[97824]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T23:57:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:33 smithi023 ceph-mon[97824]: from='osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:33 smithi023 ceph-mon[97824]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-11-16T23:57:33.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2023-11-16T23:57:34.215 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi156 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi156 to 4292M 2023-11-16T23:57:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:34.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:34.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:34.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] boot 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:34 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T23:57:34.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi156 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi156 to 4292M 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:34.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] boot 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T23:57:34.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:34 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi156 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi156 to 4292M 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:34.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: osd.10 [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] boot 2023-11-16T23:57:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: osdmap e71: 11 total, 11 up, 11 in 2023-11-16T23:57:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:34 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T23:57:35.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:35 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:57:35.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:35 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:57:35.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:35 smithi173 ceph-mon[131534]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T23:57:35.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:35 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:57:35.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:35 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:57:35.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:35 smithi156 ceph-mon[131105]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T23:57:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:35 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:57:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:35 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:57:35.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:35 smithi023 ceph-mon[97824]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 -- 172.21.15.173:0/1981436741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 msgr2=0x7fe8e00ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1981436741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00ff5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009990 tx=0x7fe8d0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 -- 172.21.15.173:0/1981436741 shutdown_connections 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1981436741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8e0100d70 0x7fe8e0107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1981436741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00ff5f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1981436741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e01003e0 0x7fe8e0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 -- 172.21.15.173:0/1981436741 >> 172.21.15.173:0/1981436741 conn(0x7fe8e00fa7f0 msgr2=0x7fe8e00fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 -- 172.21.15.173:0/1981436741 shutdown_connections 2023-11-16T23:57:35.522 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.521+0000 7fe8e8473700 1 -- 172.21.15.173:0/1981436741 wait complete. 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 Processor -- start 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 -- start start 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8e01003e0 0x7fe8e019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e0100d70 0x7fe8e019d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e019db30 con 0x7fe8e01003e0 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e019dca0 con 0x7fe8e0100d70 2023-11-16T23:57:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e8473700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e019de60 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51464/0 (socket says 172.21.15.173:51464) 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 -- 172.21.15.173:0/1199905786 learned_addr learned my addr 172.21.15.173:0/1199905786 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e6a10700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e0100d70 0x7fe8e019d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 -- 172.21.15.173:0/1199905786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e0100d70 msgr2=0x7fe8e019d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e0100d70 0x7fe8e019d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 -- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8e01003e0 msgr2=0x7fe8e019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8e01003e0 0x7fe8e019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 -- 172.21.15.173:0/1199905786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00095f0 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.522+0000 7fe8e620f700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00095c0 tx=0x7fe8d000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.523+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d001d070 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.523+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8d000fb80 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.525 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.523+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e010ab30 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.526 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.523+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0017650 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.526 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.523+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e010a9a0 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.526 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.524+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e01067e0 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.526 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.525+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe8d0022470 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.527 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.525+0000 7fe8d77fe700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:35.527 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.525+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fe8d00cd840 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.527 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.526+0000 7fe8e5a0e700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:35.527 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.526+0000 7fe8e5a0e700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc005950 tx=0x7fe8dc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:35.530 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.529+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe8d0097650 con 0x7fe8e00ff1e0 2023-11-16T23:57:35.724 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:35.724+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8e0061b70 con 0x7fe8cc0a1150 2023-11-16T23:57:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:36 smithi156 ceph-mon[131105]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:36 smithi156 ceph-mon[131105]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:36 smithi173 ceph-mon[131534]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:36 smithi173 ceph-mon[131534]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:36.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:36 smithi023 ceph-mon[97824]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:36 smithi023 ceph-mon[97824]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T23:57:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T23:57:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T23:57:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:37.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:37 smithi173 ceph-mon[131534]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:37.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:37 smithi173 ceph-mon[131534]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T23:57:37.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:37 smithi156 ceph-mon[131105]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:37.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:37 smithi156 ceph-mon[131105]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T23:57:37.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:37 smithi023 ceph-mon[97824]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:57:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:37 smithi023 ceph-mon[97824]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T23:57:38.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:38 smithi173 ceph-mon[131534]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:38.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:38 smithi173 ceph-mon[131534]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T23:57:38.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:38 smithi156 ceph-mon[131105]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:38.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:38 smithi156 ceph-mon[131105]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T23:57:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:38 smithi023 ceph-mon[97824]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:38 smithi023 ceph-mon[97824]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T23:57:39.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/3694579799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]': finished 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.173:0/1170087437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/3694579799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]': finished 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T23:57:39.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:39.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.173:0/1170087437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:39.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/3694579799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]: dispatch 2023-11-16T23:57:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f38c6af8-9481-4757-9973-dba7aa1a03b1"}]': finished 2023-11-16T23:57:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: osdmap e76: 12 total, 11 up, 12 in 2023-11-16T23:57:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.173:0/1170087437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T23:57:40.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:40 smithi173 ceph-mon[131534]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:57:40.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:40 smithi156 ceph-mon[131105]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:57:40.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:40 smithi023 ceph-mon[97824]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T23:57:42.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:42 smithi023 ceph-mon[97824]: pgmap v251: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-11-16T23:57:42.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:42 smithi173 ceph-mon[131534]: pgmap v251: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-11-16T23:57:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:42 smithi156 ceph-mon[131105]: pgmap v251: 1 pgs: 1 peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-11-16T23:57:44.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:44 smithi173 ceph-mon[131534]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:57:44.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:44 smithi023 ceph-mon[97824]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:57:44.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:44 smithi156 ceph-mon[131105]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-11-16T23:57:45.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T23:57:45.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:45.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T23:57:45.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:45.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T23:57:45.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:46 smithi173 ceph-mon[131534]: Deploying daemon osd.11 on smithi173 2023-11-16T23:57:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:46 smithi173 ceph-mon[131534]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:57:46.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:46 smithi023 ceph-mon[97824]: Deploying daemon osd.11 on smithi173 2023-11-16T23:57:46.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:46 smithi023 ceph-mon[97824]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:57:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:46 smithi156 ceph-mon[131105]: Deploying daemon osd.11 on smithi173 2023-11-16T23:57:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:46 smithi156 ceph-mon[131105]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T23:57:48.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:48 smithi173 ceph-mon[131534]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:48.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:48 smithi023 ceph-mon[97824]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:48 smithi156 ceph-mon[131105]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:49.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:49 smithi173 ceph-mon[131534]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:50.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:49 smithi023 ceph-mon[97824]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:49 smithi156 ceph-mon[131105]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:51.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:51.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:51.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:52.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:51 smithi173 ceph-mon[131534]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:52.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:51 smithi023 ceph-mon[97824]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:52.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:51 smithi156 ceph-mon[131105]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:52.992 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.992+0000 7fe8d77fe700 1 -- 172.21.15.173:0/1199905786 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe8e0061b70 con 0x7fe8cc0a1150 2023-11-16T23:57:52.992 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 msgr2=0x7fe8cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc005950 tx=0x7fe8dc0058e0 comp rx=0 tx=0).stop 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 msgr2=0x7fe8e00699a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00095c0 tx=0x7fe8d000f690 comp rx=0 tx=0).stop 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 shutdown_connections 2023-11-16T23:57:52.995 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8e0100d70 0x7fe8e019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8e00ff1e0 0x7fe8e00699a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 --2- 172.21.15.173:0/1199905786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8e01003e0 0x7fe8e019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 >> 172.21.15.173:0/1199905786 conn(0x7fe8e00fa7f0 msgr2=0x7fe8e0103920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 shutdown_connections 2023-11-16T23:57:52.996 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:57:52.994+0000 7fe8e8473700 1 -- 172.21.15.173:0/1199905786 wait complete. 2023-11-16T23:57:53.492 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.11.service 2023-11-16T23:57:53.494 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-11-16T23:57:53.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T23:57:53.526 INFO:journalctl@ceph.osd.11.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:57:53.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:53.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.956 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:53.956 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.956 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:53 smithi023 ceph-mon[97824]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:53 smithi156 ceph-mon[131105]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:53.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-mon[131534]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:54.073 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 16 23:57:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-16T23:57:53.810+0000 7fc6d2b00640 -1 osd.11 0 log_to_monitors true 2023-11-16T23:57:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:57:54.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:54 smithi023 ceph-mon[97824]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:54.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:54 smithi023 ceph-mon[97824]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:54.962 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:54 smithi173 ceph-mon[131534]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:54.962 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:54 smithi173 ceph-mon[131534]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:54 smithi156 ceph-mon[131105]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:55.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:54 smithi156 ceph-mon[131105]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T23:57:55.856 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-16T23:57:55.696+0000 7fc6c8177700 -1 osd.11 0 waiting for initial osdmap 2023-11-16T23:57:55.857 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-16T23:57:55.710+0000 7fc6c0765700 -1 osd.11 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:55.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:55 smithi173 ceph-mon[131534]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 -- 172.21.15.23:0/4170528980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b0101e80 msgr2=0x7fe9b0108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 --2- 172.21.15.23:0/4170528980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b0101e80 0x7fe9b0108740 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b0a0 tx=0x7fe9a800b3b0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 -- 172.21.15.23:0/4170528980 shutdown_connections 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 --2- 172.21.15.23:0/4170528980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b0101e80 0x7fe9b0108740 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 --2- 172.21.15.23:0/4170528980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01002f0 0x7fe9b0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 --2- 172.21.15.23:0/4170528980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b01014f0 0x7fe9b0101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 -- 172.21.15.23:0/4170528980 >> 172.21.15.23:0/4170528980 conn(0x7fe9b00fb900 msgr2=0x7fe9b00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 -- 172.21.15.23:0/4170528980 shutdown_connections 2023-11-16T23:57:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.909+0000 7fe9b494a700 1 -- 172.21.15.23:0/4170528980 wait complete. 2023-11-16T23:57:55.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.910+0000 7fe9b494a700 1 Processor -- start 2023-11-16T23:57:55.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.910+0000 7fe9b494a700 1 -- start start 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01014f0 0x7fe9b0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 0x7fe9b019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b019fee0 con 0x7fe9b01002f0 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a0020 con 0x7fe9b01014f0 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9b494a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a01b0 con 0x7fe9b0101e80 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9ae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9ae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39364/0 (socket says 172.21.15.23:39364) 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9ae59c700 1 -- 172.21.15.23:0/3364974015 learned_addr learned my addr 172.21.15.23:0/3364974015 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9aed9d700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 0x7fe9b019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:55.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.911+0000 7fe9add9b700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01014f0 0x7fe9b0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:55.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 -- 172.21.15.23:0/3364974015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01014f0 msgr2=0x7fe9b0198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:55.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01014f0 0x7fe9b0198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 -- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 msgr2=0x7fe9b0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 -- 172.21.15.23:0/3364974015 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8009d00 con 0x7fe9b0101e80 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9ae59c700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9aed9d700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 0x7fe9b019f910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b070 tx=0x7fe9a800bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a800e040 con 0x7fe9b0101e80 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b010bcb0 con 0x7fe9b0101e80 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b010bb20 con 0x7fe9b0101e80 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9a80094c0 con 0x7fe9b0101e80 2023-11-16T23:57:55.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.912+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a80052a0 con 0x7fe9b0101e80 2023-11-16T23:57:55.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.913+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0066ea0 con 0x7fe9b0101e80 2023-11-16T23:57:55.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.915+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe9a801d470 con 0x7fe9b0101e80 2023-11-16T23:57:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.916+0000 7fe99f7fe700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 0x7fe9980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:57:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.917+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fe9a80cd4e0 con 0x7fe9b0101e80 2023-11-16T23:57:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.917+0000 7fe9ae59c700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 0x7fe9980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:57:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.917+0000 7fe9ae59c700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 0x7fe9980a3600 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0007fd0 tx=0x7fe9a0007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:57:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:55.918+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe9a80971e0 con 0x7fe9b0101e80 2023-11-16T23:57:56.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T23:57:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T23:57:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:56.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:55 smithi023 ceph-mon[97824]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:56.106 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T23:57:56.460 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T23:57:56.460 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:56.460 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: from='osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:56.460 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2023-11-16T23:57:56.460 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:55 smithi156 ceph-mon[131105]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 69 MiB used, 983 GiB / 983 GiB avail 2023-11-16T23:57:56.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.141+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe9b01a0c70 con 0x7fe9b0101e80 2023-11-16T23:57:56.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.141+0000 7fe99f7fe700 1 -- 172.21.15.23:0/3364974015 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe9a80253b0 con 0x7fe9b0101e80 2023-11-16T23:57:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:57:56.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 msgr2=0x7fe9980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:56.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 0x7fe9980a3600 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0007fd0 tx=0x7fe9a0007480 comp rx=0 tx=0).stop 2023-11-16T23:57:56.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 msgr2=0x7fe9b019f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:57:56.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 0x7fe9b019f910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b070 tx=0x7fe9a800bce0 comp rx=0 tx=0).stop 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 shutdown_connections 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe9980a1150 0x7fe9980a3600 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9b0101e80 0x7fe9b019f910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9b01002f0 0x7fe9b0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 --2- 172.21.15.23:0/3364974015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9b01014f0 0x7fe9b0198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 >> 172.21.15.23:0/3364974015 conn(0x7fe9b00fb900 msgr2=0x7fe9b00fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 shutdown_connections 2023-11-16T23:57:56.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:57:56.144+0000 7fe9b494a700 1 -- 172.21.15.23:0/3364974015 wait complete. 2023-11-16T23:57:56.650 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"num_osds":12,"num_up_osds":11,"osd_up_since":1700179053,"num_in_osds":12,"osd_in_since":1700179058,"num_remapped_pgs":0} 2023-11-16T23:57:57.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: Detected new or changed devices on smithi173 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi173 to 4292M 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:56 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3364974015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:57:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:57:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T23:57:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: Detected new or changed devices on smithi173 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi173 to 4292M 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:56 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3364974015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: Detected new or changed devices on smithi173 2023-11-16T23:57:57.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.197 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T23:57:57.198 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi173 to 4292M 2023-11-16T23:57:57.198 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.198 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:57:57.198 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:57:57.198 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.200 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:57:57.200 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:57:57.201 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:57:57.201 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:56 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3364974015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:57:57.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T23:57:58.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: purged_snaps scrub starts 2023-11-16T23:57:58.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: purged_snaps scrub ok 2023-11-16T23:57:58.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] boot 2023-11-16T23:57:58.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T23:57:58.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:58.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:57 smithi023 ceph-mon[97824]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:57:58.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: purged_snaps scrub starts 2023-11-16T23:57:58.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: purged_snaps scrub ok 2023-11-16T23:57:58.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] boot 2023-11-16T23:57:58.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T23:57:58.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:58.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:57 smithi173 ceph-mon[131534]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: purged_snaps scrub starts 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: purged_snaps scrub ok 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: osd.11 [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] boot 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: osdmap e79: 12 total, 12 up, 12 in 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T23:57:58.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:57 smithi156 ceph-mon[131105]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:57:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:57:58.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:58 smithi023 ceph-mon[97824]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T23:57:59.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:58 smithi173 ceph-mon[131534]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T23:57:59.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:58 smithi156 ceph-mon[131105]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T23:57:59.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:57:59 smithi023 ceph-mon[97824]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:00.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:57:59 smithi173 ceph-mon[131534]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:00.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:57:59 smithi156 ceph-mon[131105]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 -- 172.21.15.23:0/3653861320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 msgr2=0x7f890c101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 --2- 172.21.15.23:0/3653861320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c101940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009990 tx=0x7f88fc009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 -- 172.21.15.23:0/3653861320 shutdown_connections 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 --2- 172.21.15.23:0/3653861320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 0x7f890c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 --2- 172.21.15.23:0/3653861320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 0x7f890c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 --2- 172.21.15.23:0/3653861320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c101940 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 -- 172.21.15.23:0/3653861320 >> 172.21.15.23:0/3653861320 conn(0x7f890c0fb900 msgr2=0x7f890c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 -- 172.21.15.23:0/3653861320 shutdown_connections 2023-11-16T23:58:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.115+0000 7f8913714700 1 -- 172.21.15.23:0/3653861320 wait complete. 2023-11-16T23:58:00.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 Processor -- start 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 -- start start 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 0x7f890c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 0x7f890c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.116+0000 7f8913714700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c19fdf0 con 0x7f890c1014f0 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8913714700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c19ff30 con 0x7f890c1002f0 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8913714700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1a00c0 con 0x7f890c101e80 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f89114b0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 0x7f890c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39388/0 (socket says 172.21.15.23:39388) 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 -- 172.21.15.23:0/1476939276 learned_addr learned my addr 172.21.15.23:0/1476939276 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:00.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8911cb1700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 0x7f890c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:00.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 -- 172.21.15.23:0/1476939276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 msgr2=0x7f890c19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:00.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 0x7f890c19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 -- 172.21.15.23:0/1476939276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 msgr2=0x7f890c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:00.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 0x7f890c198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 -- 172.21.15.23:0/1476939276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0095f0 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.117+0000 7f8910caf700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005910 tx=0x7f88fc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.118+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01d070 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.118+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88fc022470 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.118+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc017650 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.118+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c10bc90 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.118+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c10bb00 con 0x7f890c1014f0 2023-11-16T23:58:00.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.119+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c1929f0 con 0x7f890c1014f0 2023-11-16T23:58:00.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.121+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f88fc022850 con 0x7f890c1014f0 2023-11-16T23:58:00.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.122+0000 7f89027fc700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 0x7f88f80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:00.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.122+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88fc0cd8e0 con 0x7f890c1014f0 2023-11-16T23:58:00.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.123+0000 7f89114b0700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 0x7f88f80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:00.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.123+0000 7f89114b0700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 0x7f88f80a3680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f890c101350 tx=0x7f8908009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.127+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f88fc0974c0 con 0x7f890c1014f0 2023-11-16T23:58:00.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.336+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f890c02cfe0 con 0x7f890c1014f0 2023-11-16T23:58:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.336+0000 7f89027fc700 1 -- 172.21.15.23:0/1476939276 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f88fc09aae0 con 0x7f890c1014f0 2023-11-16T23:58:00.337 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.338+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 msgr2=0x7f88f80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.338+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 0x7f88f80a3680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f890c101350 tx=0x7f8908009260 comp rx=0 tx=0).stop 2023-11-16T23:58:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.338+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 msgr2=0x7f890c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.338+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005910 tx=0x7f88fc0057b0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 shutdown_connections 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88f80a11d0 0x7f88f80a3680 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f890c101e80 0x7f890c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f890c1002f0 0x7f890c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 --2- 172.21.15.23:0/1476939276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f890c1014f0 0x7f890c1987a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 >> 172.21.15.23:0/1476939276 conn(0x7f890c0fb900 msgr2=0x7f890c106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 shutdown_connections 2023-11-16T23:58:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:00.339+0000 7f8913714700 1 -- 172.21.15.23:0/1476939276 wait complete. 2023-11-16T23:58:00.911 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1700179076,"num_in_osds":12,"osd_in_since":1700179058,"num_remapped_pgs":0} 2023-11-16T23:58:00.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T23:58:00.975 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:00 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1476939276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:58:01.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:00 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1476939276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:58:01.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:00 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1476939276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T23:58:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:58:02.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:01 smithi023 ceph-mon[97824]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:02.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:01 smithi173 ceph-mon[131534]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:02.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:01 smithi156 ceph-mon[131105]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:03.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 -- 172.21.15.23:0/3684837971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174100370 msgr2=0x7f8174100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:03.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 --2- 172.21.15.23:0/3684837971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174100370 0x7f8174100740 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f81700099c0 tx=0x7f8170009cd0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 -- 172.21.15.23:0/3684837971 shutdown_connections 2023-11-16T23:58:03.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 --2- 172.21.15.23:0/3684837971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174105410 0x7f817410cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 --2- 172.21.15.23:0/3684837971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174100370 0x7f8174100740 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 --2- 172.21.15.23:0/3684837971 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100d10 0x7f8174104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 -- 172.21.15.23:0/3684837971 >> 172.21.15.23:0/3684837971 conn(0x7f81740fbbd0 msgr2=0x7f81740fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:03.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 -- 172.21.15.23:0/3684837971 shutdown_connections 2023-11-16T23:58:03.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.399+0000 7f817ae97700 1 -- 172.21.15.23:0/3684837971 wait complete. 2023-11-16T23:58:03.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.400+0000 7f817ae97700 1 Processor -- start 2023-11-16T23:58:03.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 -- start start 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174100d10 0x7f81741967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 0x7f817419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419b230 con 0x7f8174105410 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419b3a0 con 0x7f8174100370 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817ae97700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419b560 con 0x7f8174100d10 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f8179e95700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f8179e95700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57028/0 (socket says 172.21.15.23:57028) 2023-11-16T23:58:03.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f8179e95700 1 -- 172.21.15.23:0/2773531429 learned_addr learned my addr 172.21.15.23:0/2773531429 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f817a696700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 0x7f817419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.401+0000 7f8179694700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174100d10 0x7f81741967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 -- 172.21.15.23:0/2773531429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174100d10 msgr2=0x7f81741967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174100d10 0x7f81741967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 -- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 msgr2=0x7f817419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 0x7f817419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 -- 172.21.15.23:0/2773531429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700095f0 con 0x7f8174100370 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f817a696700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 0x7f817419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:03.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f8179e95700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8170009990 tx=0x7f817000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817001d070 con 0x7f8174100370 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81741101a0 con 0x7f8174100370 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.402+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8170005030 con 0x7f8174100370 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.403+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170004a50 con 0x7f8174100370 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.403+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8174110010 con 0x7f8174100370 2023-11-16T23:58:03.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.404+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8174110330 con 0x7f8174100370 2023-11-16T23:58:03.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.406+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8170005430 con 0x7f8174100370 2023-11-16T23:58:03.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.407+0000 7f816affd700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 0x7f81600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:03.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.407+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81700cd1f0 con 0x7f8174100370 2023-11-16T23:58:03.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.407+0000 7f8179694700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 0x7f81600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:03.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.408+0000 7f8179694700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 0x7f81600a3680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8174197890 tx=0x7f8164009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:03.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.409+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8170096dd0 con 0x7f8174100370 2023-11-16T23:58:03.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.616+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81741a32f0 con 0x7f8174100370 2023-11-16T23:58:03.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.618+0000 7f816affd700 1 -- 172.21.15.23:0/2773531429 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+18447 (secure 0 0 0) 0x7f817009a3f0 con 0x7f8174100370 2023-11-16T23:58:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:03.619 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:57:57.694103+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 msgr2=0x7f81600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 0x7f81600a3680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8174197890 tx=0x7f8164009260 comp rx=0 tx=0).stop 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 msgr2=0x7f8174196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8170009990 tx=0x7f817000f690 comp rx=0 tx=0).stop 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 shutdown_connections 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8174105410 0x7f817419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8174100370 0x7f8174196270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f81600a11d0 0x7f81600a3680 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 --2- 172.21.15.23:0/2773531429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8174100d10 0x7f81741967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 >> 172.21.15.23:0/2773531429 conn(0x7f81740fbbd0 msgr2=0x7f81740fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 shutdown_connections 2023-11-16T23:58:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:03.620+0000 7f817ae97700 1 -- 172.21.15.23:0/2773531429 wait complete. 2023-11-16T23:58:04.156 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T23:58:04.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-11-16T23:58:04.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:04 smithi156 ceph-mon[131105]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:04.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:04 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2773531429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:04.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:04 smithi173 ceph-mon[131534]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:04.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:04 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2773531429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:04.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:04 smithi023 ceph-mon[97824]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:04.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:04 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2773531429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:05.017 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:58:06.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:06 smithi173 ceph-mon[131534]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:06.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:06 smithi156 ceph-mon[131105]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:06.475 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:06 smithi023 ceph-mon[97824]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 -- 172.21.15.23:0/225946782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a01055e0 msgr2=0x7f49a010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 --2- 172.21.15.23:0/225946782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a01055e0 0x7f49a010d160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f499800b0d0 tx=0x7f499800b3e0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 -- 172.21.15.23:0/225946782 shutdown_connections 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 --2- 172.21.15.23:0/225946782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a01055e0 0x7f49a010d160 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 --2- 172.21.15.23:0/225946782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100540 0x7f49a0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 --2- 172.21.15.23:0/225946782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a0100ee0 0x7f49a0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.761+0000 7f49a4adc700 1 -- 172.21.15.23:0/225946782 >> 172.21.15.23:0/225946782 conn(0x7f49a00fbdc0 msgr2=0x7f49a00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:06.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.762+0000 7f49a4adc700 1 -- 172.21.15.23:0/225946782 shutdown_connections 2023-11-16T23:58:06.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.762+0000 7f49a4adc700 1 -- 172.21.15.23:0/225946782 wait complete. 2023-11-16T23:58:06.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.762+0000 7f49a4adc700 1 Processor -- start 2023-11-16T23:58:06.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 -- start start 2023-11-16T23:58:06.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100ee0 0x7f49a0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a01055e0 0x7f49a0073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01a48d0 con 0x7f49a0100540 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01a4a40 con 0x7f49a01055e0 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f49a4adc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01a4c00 con 0x7f49a0100ee0 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f499e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f499e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59912/0 (socket says 172.21.15.23:59912) 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f499e59c700 1 -- 172.21.15.23:0/3126709813 learned_addr learned my addr 172.21.15.23:0/3126709813 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f499ed9d700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a01055e0 0x7f49a0073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:06.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.763+0000 7f499dd9b700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100ee0 0x7f49a0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:06.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 -- 172.21.15.23:0/3126709813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100ee0 msgr2=0x7f49a0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:06.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100ee0 0x7f49a0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 -- 172.21.15.23:0/3126709813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a01055e0 msgr2=0x7f49a0073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:06.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a01055e0 0x7f49a0073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 -- 172.21.15.23:0/3126709813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998009d00 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f499e59c700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9a0 tx=0x7f499000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990009790 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f499000cbd0 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990018680 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a0110370 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.764+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a01101e0 con 0x7f49a0100540 2023-11-16T23:58:06.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.766+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0110500 con 0x7f49a0100540 2023-11-16T23:58:06.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.767+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f499000cd40 con 0x7f49a0100540 2023-11-16T23:58:06.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.769+0000 7f498f7fe700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 0x7f49880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:06.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.769+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4990014070 con 0x7f49a0100540 2023-11-16T23:58:06.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.769+0000 7f499dd9b700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 0x7f49880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:06.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.770+0000 7f499dd9b700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 0x7f49880a3680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f49a0074570 tx=0x7f4994009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:06.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.773+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4990051730 con 0x7f49a0100540 2023-11-16T23:58:06.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.978+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49a01a5740 con 0x7f49a0100540 2023-11-16T23:58:06.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.978+0000 7f498f7fe700 1 -- 172.21.15.23:0/3126709813 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4990099e00 con 0x7f49a0100540 2023-11-16T23:58:06.981 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-16T23:58:06.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 msgr2=0x7f49880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:06.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 0x7f49880a3680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f49a0074570 tx=0x7f4994009260 comp rx=0 tx=0).stop 2023-11-16T23:58:06.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 msgr2=0x7f49a0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9a0 tx=0x7f499000ed60 comp rx=0 tx=0).stop 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 shutdown_connections 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49a01055e0 0x7f49a0073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a0100540 0x7f49a0072df0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49880a11d0 0x7f49880a3680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 --2- 172.21.15.23:0/3126709813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49a0100ee0 0x7f49a0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 >> 172.21.15.23:0/3126709813 conn(0x7f49a00fbdc0 msgr2=0x7f49a00fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.980+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 shutdown_connections 2023-11-16T23:58:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:06.981+0000 7f49a4adc700 1 -- 172.21.15.23:0/3126709813 wait complete. 2023-11-16T23:58:07.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:07 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3126709813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:07.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:07 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3126709813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:07.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:07 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3126709813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.a on smithi023 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.d on smithi023 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.g on smithi023 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.j on smithi023 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.b on smithi156 2023-11-16T23:58:07.504 INFO:tasks.cephadm:Adding mds.e on smithi156 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.h on smithi156 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.k on smithi156 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.c on smithi173 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.f on smithi173 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.i on smithi173 2023-11-16T23:58:07.505 INFO:tasks.cephadm:Adding mds.l on smithi173 2023-11-16T23:58:07.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l' 2023-11-16T23:58:08.161 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T23:58:08.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:08 smithi156 ceph-mon[131105]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:08.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:08 smithi173 ceph-mon[131534]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:08.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:08 smithi023 ceph-mon[97824]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 -- 172.21.15.173:0/2272009548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164101ea0 msgr2=0x7f5164108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 --2- 172.21.15.173:0/2272009548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164101ea0 0x7f5164108760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f515c00b0a0 tx=0x7f515c00b3b0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 -- 172.21.15.173:0/2272009548 shutdown_connections 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 --2- 172.21.15.173:0/2272009548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164101ea0 0x7f5164108760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 --2- 172.21.15.173:0/2272009548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164100310 0x7f5164100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 --2- 172.21.15.173:0/2272009548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 0x7f5164101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 -- 172.21.15.173:0/2272009548 >> 172.21.15.173:0/2272009548 conn(0x7f51640fb960 msgr2=0x7f51640fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:09.486 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 -- 172.21.15.173:0/2272009548 shutdown_connections 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 -- 172.21.15.173:0/2272009548 wait complete. 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.482+0000 7f516a22d700 1 Processor -- start 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 -- start start 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 0x7f51641966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164101ea0 0x7f516419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516419dd20 con 0x7f5164101ea0 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516419de60 con 0x7f5164101510 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f516a22d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516419dff0 con 0x7f5164100310 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:09.487 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51582/0 (socket says 172.21.15.173:51582) 2023-11-16T23:58:09.533 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 -- 172.21.15.173:0/28131785 learned_addr learned my addr 172.21.15.173:0/28131785 (peer_addr_for_me v2:172.21.15.173:0/0) 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f5162ffd700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 0x7f51641966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f5163fff700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164101ea0 0x7f516419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 -- 172.21.15.173:0/28131785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 msgr2=0x7f51641966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 0x7f51641966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 -- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164101ea0 msgr2=0x7f516419d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164101ea0 0x7f516419d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 -- 172.21.15.173:0/28131785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c009d00 con 0x7f5164100310 2023-11-16T23:58:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.483+0000 7f51637fe700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f515400b530 tx=0x7f515400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.484+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154011830 con 0x7f5164100310 2023-11-16T23:58:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.484+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516410bd00 con 0x7f5164100310 2023-11-16T23:58:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.484+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5154011e50 con 0x7f5164100310 2023-11-16T23:58:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.484+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515400f2e0 con 0x7f5164100310 2023-11-16T23:58:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.484+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516410bb70 con 0x7f5164100310 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.485+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5164107810 con 0x7f5164100310 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.486+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5154010580 con 0x7f5164100310 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.486+0000 7f5160ff9700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 0x7f514c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.486+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f51540cbeb0 con 0x7f5164100310 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.487+0000 7f5162ffd700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 0x7f514c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.487+0000 7f5162ffd700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 0x7f514c0a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5158005950 tx=0x7f515800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:09.537 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.490+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5154095aa0 con 0x7f5164100310 2023-11-16T23:58:09.687 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.685+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f5164061c10 con 0x7f514c0a1150 2023-11-16T23:58:09.699 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.698+0000 7f5160ff9700 1 -- 172.21.15.173:0/28131785 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5164061c10 con 0x7f514c0a1150 2023-11-16T23:58:09.699 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mds.all update... 2023-11-16T23:58:09.703 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.699+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 msgr2=0x7f514c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 0x7f514c0a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5158005950 tx=0x7f515800b410 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 msgr2=0x7f5164196190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f515400b530 tx=0x7f515400b840 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 shutdown_connections 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5164101ea0 0x7f516419d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f514c0a1150 0x7f514c0a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5164100310 0x7f5164196190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 --2- 172.21.15.173:0/28131785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5164101510 0x7f51641966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 >> 172.21.15.173:0/28131785 conn(0x7f51640fb960 msgr2=0x7f5164104950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:09.704 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 shutdown_connections 2023-11-16T23:58:09.705 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-16T23:58:09.700+0000 7f516a22d700 1 -- 172.21.15.173:0/28131785 wait complete. 2023-11-16T23:58:10.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:09 smithi023 ceph-mon[97824]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:09 smithi173 ceph-mon[131534]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:10.101 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.a.service 2023-11-16T23:58:10.106 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:09 smithi156 ceph-mon[131105]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:10.107 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.d.service 2023-11-16T23:58:10.153 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.g.service 2023-11-16T23:58:10.189 INFO:journalctl@ceph.mds.a.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:58:10.193 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.j.service 2023-11-16T23:58:10.228 INFO:journalctl@ceph.mds.d.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:58:10.231 DEBUG:teuthology.orchestra.run.smithi156:mds.b> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.b.service 2023-11-16T23:58:10.233 DEBUG:teuthology.orchestra.run.smithi156:mds.e> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.e.service 2023-11-16T23:58:10.264 INFO:journalctl@ceph.mds.b.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:58:10.264 DEBUG:teuthology.orchestra.run.smithi156:mds.h> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.h.service 2023-11-16T23:58:10.266 INFO:journalctl@ceph.mds.g.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:58:10.290 INFO:journalctl@ceph.mds.e.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:58:10.291 DEBUG:teuthology.orchestra.run.smithi156:mds.k> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.k.service 2023-11-16T23:58:10.302 INFO:journalctl@ceph.mds.j.smithi023.stdout:-- Logs begin at Thu 2023-11-16 23:32:51 UTC. -- 2023-11-16T23:58:10.316 INFO:journalctl@ceph.mds.h.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:58:10.317 DEBUG:teuthology.orchestra.run.smithi173:mds.c> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service 2023-11-16T23:58:10.321 DEBUG:teuthology.orchestra.run.smithi173:mds.f> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.f.service 2023-11-16T23:58:10.343 INFO:journalctl@ceph.mds.k.smithi156.stdout:-- Logs begin at Thu 2023-11-16 23:27:50 UTC. -- 2023-11-16T23:58:10.351 INFO:journalctl@ceph.mds.c.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:58:10.351 DEBUG:teuthology.orchestra.run.smithi173:mds.i> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.i.service 2023-11-16T23:58:10.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:58:10.378 DEBUG:teuthology.orchestra.run.smithi173:mds.l> sudo journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service 2023-11-16T23:58:10.403 INFO:journalctl@ceph.mds.i.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:58:10.404 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-11-16T23:58:10.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:58:10.429 INFO:journalctl@ceph.mds.l.smithi173.stdout:-- Logs begin at Thu 2023-11-16 23:27:37 UTC. -- 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 -- 172.21.15.23:0/3711882879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1014a0 msgr2=0x7fa26c1018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3711882879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1014a0 0x7fa26c1018f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa2540098f0 tx=0x7fa254009c00 comp rx=0 tx=0).stop 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 -- 172.21.15.23:0/3711882879 shutdown_connections 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3711882879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3711882879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1002b0 0x7fa26c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3711882879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1014a0 0x7fa26c1018f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 -- 172.21.15.23:0/3711882879 >> 172.21.15.23:0/3711882879 conn(0x7fa26c0fb890 msgr2=0x7fa26c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:10.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 -- 172.21.15.23:0/3711882879 shutdown_connections 2023-11-16T23:58:10.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.624+0000 7fa2717ff700 1 -- 172.21.15.23:0/3711882879 wait complete. 2023-11-16T23:58:10.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.625+0000 7fa2717ff700 1 Processor -- start 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.625+0000 7fa2717ff700 1 -- start start 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1002b0 0x7fa26c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1014a0 0x7fa26c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c19fd40 con 0x7fa26c101e30 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c19fe80 con 0x7fa26c1014a0 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa2717ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1a0010 con 0x7fa26c1002b0 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1002b0 0x7fa26c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59938/0 (socket says 172.21.15.23:59938) 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 -- 172.21.15.23:0/3101089836 learned_addr learned my addr 172.21.15.23:0/3101089836 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:10.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa263fff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1014a0 0x7fa26c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:10.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 -- 172.21.15.23:0/3101089836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1002b0 msgr2=0x7fa26c1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:10.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1002b0 0x7fa26c1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 -- 172.21.15.23:0/3101089836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1014a0 msgr2=0x7fa26c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:10.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1014a0 0x7fa26c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.626+0000 7fa26b7fe700 1 -- 172.21.15.23:0/3101089836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254009520 con 0x7fa26c101e30 2023-11-16T23:58:10.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa26b7fe700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa26400c110 tx=0x7fa26400c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:10.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264017070 con 0x7fa26c101e30 2023-11-16T23:58:10.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa26400e050 con 0x7fa26c101e30 2023-11-16T23:58:10.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c10bc10 con 0x7fa26c101e30 2023-11-16T23:58:10.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640078f0 con 0x7fa26c101e30 2023-11-16T23:58:10.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.627+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c10ba80 con 0x7fa26c101e30 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.628+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c1928a0 con 0x7fa26c101e30 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.630+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa264007a50 con 0x7fa26c101e30 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.631+0000 7fa268ff9700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 0x7fa2580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.632+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2640cbf00 con 0x7fa26c101e30 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.632+0000 7fa26affd700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 0x7fa2580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.632+0000 7fa26affd700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 0x7fa2580a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa26c101300 tx=0x7fa25c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:10.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.636+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa264095a50 con 0x7fa26c101e30 2023-11-16T23:58:10.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:10.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:10.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:10.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.850+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa26c02cfe0 con 0x7fa26c101e30 2023-11-16T23:58:10.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.851+0000 7fa268ff9700 1 -- 172.21.15.23:0/3101089836 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+18447 (secure 0 0 0) 0x7fa264099070 con 0x7fa26c101e30 2023-11-16T23:58:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:10.852 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:57:57.694103+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:10.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 msgr2=0x7fa2580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:10.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 0x7fa2580a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa26c101300 tx=0x7fa25c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:10.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 msgr2=0x7fa26c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa26400c110 tx=0x7fa26400c420 comp rx=0 tx=0).stop 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 shutdown_connections 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa26c101e30 0x7fa26c19f770 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa2580a1150 0x7fa2580a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa26c1002b0 0x7fa26c1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 --2- 172.21.15.23:0/3101089836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c1014a0 0x7fa26c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.853+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 >> 172.21.15.23:0/3101089836 conn(0x7fa26c0fb890 msgr2=0x7fa26c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:10.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.854+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 shutdown_connections 2023-11-16T23:58:10.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:10.854+0000 7fa2717ff700 1 -- 172.21.15.23:0/3101089836 wait complete. 2023-11-16T23:58:10.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T23:58:10.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:11.034 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:11.035 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 -- 172.21.15.23:0/858888013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 msgr2=0x7fe38c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 --2- 172.21.15.23:0/858888013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1006e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe388009990 tx=0x7fe388009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 -- 172.21.15.23:0/858888013 shutdown_connections 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 --2- 172.21.15.23:0/858888013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 0x7fe38c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 --2- 172.21.15.23:0/858888013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1006e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 --2- 172.21.15.23:0/858888013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 0x7fe38c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 -- 172.21.15.23:0/858888013 >> 172.21.15.23:0/858888013 conn(0x7fe38c0fb8b0 msgr2=0x7fe38c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 -- 172.21.15.23:0/858888013 shutdown_connections 2023-11-16T23:58:11.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.083+0000 7fe3945ff700 1 -- 172.21.15.23:0/858888013 wait complete. 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.084+0000 7fe3945ff700 1 Processor -- start 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 -- start start 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 0x7fe38c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 0x7fe38c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c19fe40 con 0x7fe38c1002d0 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c19ff80 con 0x7fe38c1014c0 2023-11-16T23:58:11.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe3945ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1a0110 con 0x7fe38c101e50 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe39239b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe39239b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58354/0 (socket says 172.21.15.23:58354) 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe39239b700 1 -- 172.21.15.23:0/2606706664 learned_addr learned my addr 172.21.15.23:0/2606706664 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe392b9c700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 0x7fe38c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.085+0000 7fe391b9a700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 0x7fe38c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 -- 172.21.15.23:0/2606706664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 msgr2=0x7fe38c19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 0x7fe38c19f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 -- 172.21.15.23:0/2606706664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 msgr2=0x7fe38c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 0x7fe38c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 -- 172.21.15.23:0/2606706664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880095f0 con 0x7fe38c1002d0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe39239b700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe388004730 tx=0x7fe388004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38801d070 con 0x7fe38c1002d0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe388021410 con 0x7fe38c1002d0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c10bca0 con 0x7fe38c1002d0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388017650 con 0x7fe38c1002d0 2023-11-16T23:58:11.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.086+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c10bb10 con 0x7fe38c1002d0 2023-11-16T23:58:11.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.087+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c1a0710 con 0x7fe38c1002d0 2023-11-16T23:58:11.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.089+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe388021580 con 0x7fe38c1002d0 2023-11-16T23:58:11.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.090+0000 7fe3837fe700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 0x7fe3780a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.091+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3880cd870 con 0x7fe38c1002d0 2023-11-16T23:58:11.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.091+0000 7fe391b9a700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 0x7fe3780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.091+0000 7fe391b9a700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 0x7fe3780a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009c60 tx=0x7fe37c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:11.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.094+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe388097450 con 0x7fe38c1002d0 2023-11-16T23:58:11.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:58:11.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2023-11-16T23:58:11.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:11.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.304+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe38c1a0db0 con 0x7fe38c1002d0 2023-11-16T23:58:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.305+0000 7fe3837fe700 1 -- 172.21.15.23:0/2606706664 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe38809aa70 con 0x7fe38c1002d0 2023-11-16T23:58:11.306 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-16T23:58:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 msgr2=0x7fe3780a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 0x7fe3780a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009c60 tx=0x7fe37c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 msgr2=0x7fe38c1982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe388004730 tx=0x7fe388004760 comp rx=0 tx=0).stop 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 shutdown_connections 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3780a11d0 0x7fe3780a3680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe38c101e50 0x7fe38c19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe38c1002d0 0x7fe38c1982c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 --2- 172.21.15.23:0/2606706664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe38c1014c0 0x7fe38c198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.307+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 >> 172.21.15.23:0/2606706664 conn(0x7fe38c0fb8b0 msgr2=0x7fe38c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.308+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 shutdown_connections 2023-11-16T23:58:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.308+0000 7fe3945ff700 1 -- 172.21.15.23:0/2606706664 wait complete. 2023-11-16T23:58:11.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 -- 172.21.15.23:0/2083060235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101480 msgr2=0x7f65f81018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 --2- 172.21.15.23:0/2083060235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101480 0x7f65f81018d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009990 tx=0x7f65e0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 -- 172.21.15.23:0/2083060235 shutdown_connections 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 --2- 172.21.15.23:0/2083060235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8101e10 0x7f65f81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 --2- 172.21.15.23:0/2083060235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8100290 0x7f65f81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 --2- 172.21.15.23:0/2083060235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101480 0x7f65f81018d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.531+0000 7f65fcdff700 1 -- 172.21.15.23:0/2083060235 >> 172.21.15.23:0/2083060235 conn(0x7f65f80fb870 msgr2=0x7f65f80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.532+0000 7f65fcdff700 1 -- 172.21.15.23:0/2083060235 shutdown_connections 2023-11-16T23:58:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.532+0000 7f65fcdff700 1 -- 172.21.15.23:0/2083060235 wait complete. 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 Processor -- start 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 -- start start 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 0x7f65f8195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8101480 0x7f65f81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819db10 con 0x7f65f8101e10 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819dc50 con 0x7f65f8100290 2023-11-16T23:58:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65fcdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f819dde0 con 0x7f65f8101480 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65f6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65f659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 0x7f65f8195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65f6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58374/0 (socket says 172.21.15.23:58374) 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65f6d9d700 1 -- 172.21.15.23:0/1090118505 learned_addr learned my addr 172.21.15.23:0/1090118505 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.533+0000 7f65effff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8101480 0x7f65f81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 -- 172.21.15.23:0/1090118505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8101480 msgr2=0x7f65f81964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8101480 0x7f65f81964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 -- 172.21.15.23:0/1090118505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 msgr2=0x7f65f8195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 0x7f65f8195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 -- 172.21.15.23:0/1090118505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00095f0 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f659c700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 0x7f65f8195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65f6d9d700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65f000ef30 tx=0x7f65f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f0018070 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65f000f040 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f810bc50 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f000cd00 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.534+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f810bac0 con 0x7f65f8101e10 2023-11-16T23:58:11.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.535+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f819e4c0 con 0x7f65f8101e10 2023-11-16T23:58:11.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.537+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f65f0003a00 con 0x7f65f8101e10 2023-11-16T23:58:11.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.539+0000 7f65ef7fe700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 0x7f65e40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:11.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.539+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f65f00cbfb0 con 0x7f65f8101e10 2023-11-16T23:58:11.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.539+0000 7f65f659c700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 0x7f65e40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:11.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.540+0000 7f65f659c700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 0x7f65e40a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f65f81012e0 tx=0x7f65e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:11.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.543+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f65f0095b90 con 0x7f65f8101e10 2023-11-16T23:58:11.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.802+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65f81071d0 con 0x7f65f8101e10 2023-11-16T23:58:11.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.802+0000 7f65ef7fe700 1 -- 172.21.15.23:0/1090118505 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f65f00991b0 con 0x7f65f8101e10 2023-11-16T23:58:11.803 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:11.803 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 msgr2=0x7f65e40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 0x7f65e40a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f65f81012e0 tx=0x7f65e8009260 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 msgr2=0x7f65f819d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65f000ef30 tx=0x7f65f000c3a0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 shutdown_connections 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65f8101e10 0x7f65f819d540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65f8100290 0x7f65f8195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f65e40a11d0 0x7f65e40a3680 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 --2- 172.21.15.23:0/1090118505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65f8101480 0x7f65f81964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:11.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 >> 172.21.15.23:0/1090118505 conn(0x7f65f80fb870 msgr2=0x7f65f80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:11.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 shutdown_connections 2023-11-16T23:58:11.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:11.805+0000 7f65fcdff700 1 -- 172.21.15.23:0/1090118505 wait complete. 2023-11-16T23:58:11.807 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2023-11-16T23:58:11.925 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:11 smithi173 ceph-mon[131534]: Deploying daemon mds.l on smithi173 2023-11-16T23:58:11.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:11 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3101089836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:11.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:11 smithi173 ceph-mon[131534]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:11.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:11 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2606706664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:11 smithi023 ceph-mon[97824]: Deploying daemon mds.l on smithi173 2023-11-16T23:58:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:11 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3101089836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:11 smithi023 ceph-mon[97824]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:11 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2606706664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:12.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:11 smithi156 ceph-mon[131105]: Deploying daemon mds.l on smithi173 2023-11-16T23:58:12.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:11 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3101089836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:12.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:11 smithi156 ceph-mon[131105]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:12.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:11 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2606706664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:12.223 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:11 smithi173 systemd[1]: Starting Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:12.488 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:12 smithi173 podman[156662]: 2023-11-16T23:58:12.488 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mds[156702]: -- 172.21.15.173:0/2780993060 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559fed753520 con 0x559fed729000 2023-11-16T23:58:12.858 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[156678]: starting mds.l at 2023-11-16T23:58:12.858 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:12 smithi173 bash[156662]: 0b61ee55a09b517afc0360a94f50db8823bd0fd042d80c3002e610f41d2a206b 2023-11-16T23:58:12.858 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 16 23:58:12 smithi173 systemd[1]: Started Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1090118505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6832/1743311827,v1:172.21.15.173:6833/1743311827] up:boot 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: fsmap 1 up:standby 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:12.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:13.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1090118505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:13.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6832/1743311827,v1:172.21.15.173:6833/1743311827] up:boot 2023-11-16T23:58:13.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: fsmap 1 up:standby 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1090118505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:13.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6832/1743311827,v1:172.21.15.173:6833/1743311827] up:boot 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: fsmap 1 up:standby 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:13.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:13.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:14.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:13 smithi023 ceph-mon[97824]: Deploying daemon mds.k on smithi156 2023-11-16T23:58:14.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:13 smithi023 ceph-mon[97824]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:14.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:13 smithi156 ceph-mon[131105]: Deploying daemon mds.k on smithi156 2023-11-16T23:58:14.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:13 smithi156 ceph-mon[131105]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:14.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 -- 172.21.15.23:0/2205639547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 msgr2=0x7f42441018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:14.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2205639547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441018d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4238009990 tx=0x7f4238009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 -- 172.21.15.23:0/2205639547 shutdown_connections 2023-11-16T23:58:14.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2205639547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244101e10 0x7f42441086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2205639547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244100290 0x7f42441006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2205639547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441018d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 -- 172.21.15.23:0/2205639547 >> 172.21.15.23:0/2205639547 conn(0x7f42440fb870 msgr2=0x7f42440fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:14.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 -- 172.21.15.23:0/2205639547 shutdown_connections 2023-11-16T23:58:14.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.043+0000 7f424a5ff700 1 -- 172.21.15.23:0/2205639547 wait complete. 2023-11-16T23:58:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 Processor -- start 2023-11-16T23:58:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 -- start start 2023-11-16T23:58:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244101e10 0x7f424419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424419fd30 con 0x7f4244101480 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424419fe70 con 0x7f4244100290 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.044+0000 7f424a5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42441a0000 con 0x7f4244101e10 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:37454/0 (socket says 172.21.15.23:37454) 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f42437fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 -- 172.21.15.23:0/2101457981 learned_addr learned my addr 172.21.15.23:0/2101457981 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:14.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4248b9c700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244101e10 0x7f424419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 -- 172.21.15.23:0/2101457981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244101e10 msgr2=0x7f424419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244101e10 0x7f424419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 -- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 msgr2=0x7f42441986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 -- 172.21.15.23:0/2101457981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380095f0 con 0x7f4244100290 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f4243fff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f423400e970 tx=0x7f423400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234009790 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424410bc50 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f423400c7e0 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.045+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234003c70 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.046+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424410bac0 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.046+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42441928e0 con 0x7f4244100290 2023-11-16T23:58:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.049+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f423400cc50 con 0x7f4244100290 2023-11-16T23:58:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.050+0000 7f42417fa700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 0x7f422c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.050+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4234014070 con 0x7f4244100290 2023-11-16T23:58:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.050+0000 7f42437fe700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 0x7f422c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.051+0000 7f42437fe700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 0x7f422c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f423800b2e0 tx=0x7f4238005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.054+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f42340961f0 con 0x7f4244100290 2023-11-16T23:58:14.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:13 smithi173 ceph-mon[131534]: Deploying daemon mds.k on smithi156 2023-11-16T23:58:14.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:13 smithi173 ceph-mon[131534]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:14.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.308+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424402cfe0 con 0x7f4244100290 2023-11-16T23:58:14.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.309+0000 7f42417fa700 1 -- 172.21.15.23:0/2101457981 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f4234003dd0 con 0x7f4244100290 2023-11-16T23:58:14.311 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:14.311 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T23:58:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 msgr2=0x7f422c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 0x7f422c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f423800b2e0 tx=0x7f4238005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 msgr2=0x7f42441981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f423400e970 tx=0x7f423400ec80 comp rx=0 tx=0).stop 2023-11-16T23:58:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 shutdown_connections 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f422c0a1150 0x7f422c0a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4244101e10 0x7f424419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4244100290 0x7f42441981b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 --2- 172.21.15.23:0/2101457981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4244101480 0x7f42441986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 >> 172.21.15.23:0/2101457981 conn(0x7f42440fb870 msgr2=0x7f4244106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 shutdown_connections 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:14.312+0000 7f424a5ff700 1 -- 172.21.15.23:0/2101457981 wait complete. 2023-11-16T23:58:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2023-11-16T23:58:14.357 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 systemd[1]: Starting Ceph mds.k for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:14.733 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 podman[155392]: 2023-11-16T23:58:14.733 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 bash[155392]: 181380572c0dc3ed701e9863b46b52e86600aa04194371bfef9c3affea6d6f1c 2023-11-16T23:58:14.733 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 ceph-mds[155432]: -- 172.21.15.156:0/3402053968 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c9dc8f3520 con 0x55c9dc8c9000 2023-11-16T23:58:14.733 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-k[155408]: starting mds.k at 2023-11-16T23:58:15.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:14 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2101457981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:15.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:14 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2101457981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:15.107 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 16 23:58:14 smithi156 systemd[1]: Started Ceph mds.k for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:15.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:14 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2101457981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:16.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6832/2227062426,v1:172.21.15.156:6833/2227062426] up:boot 2023-11-16T23:58:16.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: fsmap 2 up:standby 2023-11-16T23:58:16.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T23:58:16.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: Deploying daemon mds.j on smithi023 2023-11-16T23:58:16.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:15 smithi023 ceph-mon[97824]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6832/2227062426,v1:172.21.15.156:6833/2227062426] up:boot 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: fsmap 2 up:standby 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: Deploying daemon mds.j on smithi023 2023-11-16T23:58:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:15 smithi173 ceph-mon[131534]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6832/2227062426,v1:172.21.15.156:6833/2227062426] up:boot 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: fsmap 2 up:standby 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: Deploying daemon mds.j on smithi023 2023-11-16T23:58:16.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:15 smithi156 ceph-mon[131105]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:16.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 -- 172.21.15.23:0/3847409484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 msgr2=0x7f3e5c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 --2- 172.21.15.23:0/3847409484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c1057c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009990 tx=0x7f3e50009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 -- 172.21.15.23:0/3847409484 shutdown_connections 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 --2- 172.21.15.23:0/3847409484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 0x7f3e5c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 --2- 172.21.15.23:0/3847409484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 --2- 172.21.15.23:0/3847409484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c1057c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 -- 172.21.15.23:0/3847409484 >> 172.21.15.23:0/3847409484 conn(0x7f3e5c0fa750 msgr2=0x7f3e5c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 -- 172.21.15.23:0/3847409484 shutdown_connections 2023-11-16T23:58:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.543+0000 7f3e617ff700 1 -- 172.21.15.23:0/3847409484 wait complete. 2023-11-16T23:58:16.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.544+0000 7f3e617ff700 1 Processor -- start 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.544+0000 7f3e617ff700 1 -- start start 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c0fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 0x7f3e5c0ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0ffaa0 con 0x7f3e5c068af0 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0ffc10 con 0x7f3e5c103360 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e617ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c1a0040 con 0x7f3e5c105e50 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e5a7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c0fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58406/0 (socket says 172.21.15.23:58406) 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e5affd700 1 -- 172.21.15.23:0/828408499 learned_addr learned my addr 172.21.15.23:0/828408499 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.545+0000 7f3e5b7fe700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 0x7f3e5c0ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:16.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 -- 172.21.15.23:0/828408499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 msgr2=0x7f3e5c0ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 0x7f3e5c0ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 -- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 msgr2=0x7f3e5c100780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 -- 172.21.15.23:0/828408499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500095f0 con 0x7f3e5c103360 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5affd700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e5a7fc700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c0fedc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b680 tx=0x7f3e50009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5001d070 con 0x7f3e5c103360 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e50021410 con 0x7f3e5c103360 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c10bc50 con 0x7f3e5c103360 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50017650 con 0x7f3e5c103360 2023-11-16T23:58:16.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.546+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c10bac0 con 0x7f3e5c103360 2023-11-16T23:58:16.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.548+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c0fc2c0 con 0x7f3e5c103360 2023-11-16T23:58:16.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.549+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3e50021810 con 0x7f3e5c103360 2023-11-16T23:58:16.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.551+0000 7f3e43fff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 0x7f3e440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:16.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.551+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3e500cd8a0 con 0x7f3e5c103360 2023-11-16T23:58:16.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.551+0000 7f3e5affd700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 0x7f3e440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:16.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.552+0000 7f3e5affd700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 0x7f3e440a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c005950 tx=0x7f3e4c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:16.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.555+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e50097480 con 0x7f3e5c103360 2023-11-16T23:58:16.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.853+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e5c02cfe0 con 0x7f3e5c103360 2023-11-16T23:58:16.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.854+0000 7f3e43fff700 1 -- 172.21.15.23:0/828408499 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f3e5009aaa0 con 0x7f3e5c103360 2023-11-16T23:58:16.854 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:16.854 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 msgr2=0x7f3e440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 0x7f3e440a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c005950 tx=0x7f3e4c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 msgr2=0x7f3e5c0fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c0fedc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b680 tx=0x7f3e50009580 comp rx=0 tx=0).stop 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 shutdown_connections 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e440a1150 0x7f3e440a3600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c105e50 0x7f3e5c0ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e5c068af0 0x7f3e5c100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 --2- 172.21.15.23:0/828408499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e5c103360 0x7f3e5c0fedc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:16.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 >> 172.21.15.23:0/828408499 conn(0x7f3e5c0fa750 msgr2=0x7f3e5c101700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:16.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 shutdown_connections 2023-11-16T23:58:16.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:16.856+0000 7f3e617ff700 1 -- 172.21.15.23:0/828408499 wait complete. 2023-11-16T23:58:16.858 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2023-11-16T23:58:17.158 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/828408499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:17.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/828408499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:17.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/828408499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:17.518 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 systemd[1]: Starting Ceph mds.j for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:17.912 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 podman[130410]: 2023-11-16T23:58:17.912 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 bash[130410]: 12182c2089fb133829552763968663f0f8718ec48cf367757796f5ef9d4f7e55 2023-11-16T23:58:17.912 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 ceph-mds[130450]: -- 172.21.15.23:0/1596666663 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c42005f520 con 0x55c420036c00 2023-11-16T23:58:17.912 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-j[130426]: starting mds.j at 2023-11-16T23:58:18.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:17 smithi023 ceph-mon[97824]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:18.270 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 16 23:58:17 smithi023 systemd[1]: Started Ceph mds.j for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:18.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:17 smithi173 ceph-mon[131534]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:18.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:17 smithi156 ceph-mon[131105]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:18.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 -- 172.21.15.23:0/2165834215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101480 msgr2=0x7f97fc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 --2- 172.21.15.23:0/2165834215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101480 0x7f97fc1018d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f97f00098f0 tx=0x7f97f0009c00 comp rx=0 tx=0).stop 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 -- 172.21.15.23:0/2165834215 shutdown_connections 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 --2- 172.21.15.23:0/2165834215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101e10 0x7f97fc1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 --2- 172.21.15.23:0/2165834215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 0x7f97fc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 --2- 172.21.15.23:0/2165834215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101480 0x7f97fc1018d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 -- 172.21.15.23:0/2165834215 >> 172.21.15.23:0/2165834215 conn(0x7f97fc0fb870 msgr2=0x7f97fc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 -- 172.21.15.23:0/2165834215 shutdown_connections 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.096+0000 7f98021ff700 1 -- 172.21.15.23:0/2165834215 wait complete. 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.097+0000 7f98021ff700 1 Processor -- start 2023-11-16T23:58:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.097+0000 7f98021ff700 1 -- start start 2023-11-16T23:58:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.097+0000 7f98021ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 0x7f97fc1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.097+0000 7f98021ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.097+0000 7f98021ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 0x7f97fc19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f98021ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc19fd60 con 0x7f97fc101480 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f98021ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc19fea0 con 0x7f97fc101e10 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f98021ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1a0030 con 0x7f97fc100290 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58454/0 (socket says 172.21.15.23:58454) 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 -- 172.21.15.23:0/2770358374 learned_addr learned my addr 172.21.15.23:0/2770358374 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97fb7fe700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 0x7f97fc1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97fbfff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 0x7f97fc19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 -- 172.21.15.23:0/2770358374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 msgr2=0x7f97fc1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 0x7f97fc1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 -- 172.21.15.23:0/2770358374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 msgr2=0x7f97fc19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 0x7f97fc19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 -- 172.21.15.23:0/2770358374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0095f0 con 0x7f97fc101480 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97fbfff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 0x7f97fc19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.098+0000 7f97faffd700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b730 tx=0x7f97f000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.099+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f001c070 con 0x7f97fc101480 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.099+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97f000f7b0 con 0x7f97fc101480 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.099+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f4009c40 con 0x7f97fc101480 2023-11-16T23:58:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.099+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0004a70 con 0x7f97fc101480 2023-11-16T23:58:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.099+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f0009520 con 0x7f97fc101480 2023-11-16T23:58:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.100+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc10ba70 con 0x7f97fc101480 2023-11-16T23:58:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.103+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f97f0020070 con 0x7f97fc101480 2023-11-16T23:58:19.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.105+0000 7f97f8ff9700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 0x7f97e40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:19.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.105+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f97f00cd990 con 0x7f97fc101480 2023-11-16T23:58:19.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.105+0000 7f97fb7fe700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 0x7f97e40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:19.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.106+0000 7f97fb7fe700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 0x7f97e40a3680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97fc1012e0 tx=0x7f97ec009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:19.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.108+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f97f0097570 con 0x7f97fc101480 2023-11-16T23:58:19.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.23:6834/1622872783,v1:172.21.15.23:6835/1622872783] up:boot 2023-11-16T23:58:19.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: fsmap 3 up:standby 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:18 smithi023 ceph-mon[97824]: Deploying daemon mds.d on smithi023 2023-11-16T23:58:19.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.23:6834/1622872783,v1:172.21.15.23:6835/1622872783] up:boot 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: fsmap 3 up:standby 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:19.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:18 smithi173 ceph-mon[131534]: Deploying daemon mds.d on smithi023 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.23:6834/1622872783,v1:172.21.15.23:6835/1622872783] up:boot 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: fsmap 3 up:standby 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:19.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:18 smithi156 ceph-mon[131105]: Deploying daemon mds.d on smithi023 2023-11-16T23:58:19.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.408+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97fc1a0a50 con 0x7f97fc101480 2023-11-16T23:58:19.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.409+0000 7f97f8ff9700 1 -- 172.21.15.23:0/2770358374 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f97f009ab90 con 0x7f97fc101480 2023-11-16T23:58:19.409 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:19.409 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 msgr2=0x7f97e40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 0x7f97e40a3680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97fc1012e0 tx=0x7f97ec009310 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 msgr2=0x7f97fc198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b730 tx=0x7f97f000b760 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 shutdown_connections 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97fc101e10 0x7f97fc19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97e40a11d0 0x7f97e40a3680 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97fc100290 0x7f97fc1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 --2- 172.21.15.23:0/2770358374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc101480 0x7f97fc198720 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 >> 172.21.15.23:0/2770358374 conn(0x7f97fc0fb870 msgr2=0x7f97fc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 shutdown_connections 2023-11-16T23:58:19.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:19.411+0000 7f98021ff700 1 -- 172.21.15.23:0/2770358374 wait complete. 2023-11-16T23:58:19.413 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2023-11-16T23:58:20.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:19 smithi023 ceph-mon[97824]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:20.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2770358374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:20.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:19 smithi173 ceph-mon[131534]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:20.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2770358374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:20.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:19 smithi156 ceph-mon[131105]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:20.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2770358374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:20.518 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 systemd[1]: Starting Ceph mds.d for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:20.961 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 podman[130798]: 2023-11-16T23:58:21.268 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 bash[130798]: 0312ec601765d051c44ed644418e4ccb0bed21aa8109d3589dbdbf696a9b3ce0 2023-11-16T23:58:21.268 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 ceph-mds[130838]: -- 172.21.15.23:0/1507018778 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564b938d5520 con 0x564b938ad000 2023-11-16T23:58:21.268 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-d[130814]: starting mds.d at 2023-11-16T23:58:21.268 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 16 23:58:20 smithi023 systemd[1]: Started Ceph mds.d for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:21.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:21.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 -- 172.21.15.23:0/3061572791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7690073550 msgr2=0x7f76900739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 --2- 172.21.15.23:0/3061572791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7690073550 0x7f76900739c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7680009990 tx=0x7f7680009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 -- 172.21.15.23:0/3061572791 shutdown_connections 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 --2- 172.21.15.23:0/3061572791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769010c860 0x7f769010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 --2- 172.21.15.23:0/3061572791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690074b30 0x7f7690072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 --2- 172.21.15.23:0/3061572791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7690073550 0x7f76900739c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 -- 172.21.15.23:0/3061572791 >> 172.21.15.23:0/3061572791 conn(0x7f76900fb870 msgr2=0x7f76900fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 -- 172.21.15.23:0/3061572791 shutdown_connections 2023-11-16T23:58:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.643+0000 7f76983ff700 1 -- 172.21.15.23:0/3061572791 wait complete. 2023-11-16T23:58:21.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.644+0000 7f76983ff700 1 Processor -- start 2023-11-16T23:58:21.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 -- start start 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 0x7f769019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 0x7f769019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76901a4170 con 0x7f769010c860 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76901a42b0 con 0x7f7690073550 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f76983ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76901a4440 con 0x7f7690074b30 2023-11-16T23:58:21.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f769699c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f769699c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52348/0 (socket says 172.21.15.23:52348) 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f769699c700 1 -- 172.21.15.23:0/3507850528 learned_addr learned my addr 172.21.15.23:0/3507850528 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.645+0000 7f769599a700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 0x7f769019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769619b700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 0x7f769019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 -- 172.21.15.23:0/3507850528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 msgr2=0x7f769019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 0x7f769019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 -- 172.21.15.23:0/3507850528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 msgr2=0x7f769019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 0x7f769019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 -- 172.21.15.23:0/3507850528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800095f0 con 0x7f769010c860 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769619b700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 0x7f769019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769599a700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 0x7f769019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f769699c700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f768800ef30 tx=0x7f768800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:21.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688018070 con 0x7f769010c860 2023-11-16T23:58:21.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768800f040 con 0x7f769010c860 2023-11-16T23:58:21.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7690112270 con 0x7f769010c860 2023-11-16T23:58:21.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800cdb0 con 0x7f769010c860 2023-11-16T23:58:21.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.646+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76901120e0 con 0x7f769010c860 2023-11-16T23:58:21.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.648+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76901023b0 con 0x7f769010c860 2023-11-16T23:58:21.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.650+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7688003a00 con 0x7f769010c860 2023-11-16T23:58:21.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.651+0000 7f76877fe700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 0x7f767c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:21.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.651+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76880cbfe0 con 0x7f769010c860 2023-11-16T23:58:21.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.651+0000 7f769619b700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 0x7f767c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:21.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.652+0000 7f769619b700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 0x7f767c0a3680 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7690074f00 tx=0x7f768c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:21.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.655+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7688095c20 con 0x7f769010c860 2023-11-16T23:58:21.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.909+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f769002cfe0 con 0x7f769010c860 2023-11-16T23:58:21.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.910+0000 7f76877fe700 1 -- 172.21.15.23:0/3507850528 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7f7688099240 con 0x7f769010c860 2023-11-16T23:58:21.910 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:21.910 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T23:58:21.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 msgr2=0x7f767c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 0x7f767c0a3680 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7690074f00 tx=0x7f768c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 msgr2=0x7f76901a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f768800ef30 tx=0x7f768800c3a0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 shutdown_connections 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f769010c860 0x7f76901a3ba0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7690073550 0x7f769019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f767c0a11d0 0x7f767c0a3680 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 --2- 172.21.15.23:0/3507850528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7690074b30 0x7f769019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 >> 172.21.15.23:0/3507850528 conn(0x7f76900fb870 msgr2=0x7f76900fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 shutdown_connections 2023-11-16T23:58:21.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:21.912+0000 7f76983ff700 1 -- 172.21.15.23:0/3507850528 wait complete. 2023-11-16T23:58:21.914 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2023-11-16T23:58:22.197 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.23:6836/4279927842,v1:172.21.15.23:6837/4279927842] up:boot 2023-11-16T23:58:22.197 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: fsmap 4 up:standby 2023-11-16T23:58:22.197 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T23:58:22.197 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: Deploying daemon mds.e on smithi156 2023-11-16T23:58:22.198 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:22 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3507850528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.23:6836/4279927842,v1:172.21.15.23:6837/4279927842] up:boot 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: fsmap 4 up:standby 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:22.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:22.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: Deploying daemon mds.e on smithi156 2023-11-16T23:58:22.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3507850528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:22.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.23:6836/4279927842,v1:172.21.15.23:6837/4279927842] up:boot 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: fsmap 4 up:standby 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: Deploying daemon mds.e on smithi156 2023-11-16T23:58:22.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:22 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3507850528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:22.857 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:22 smithi156 systemd[1]: Starting Ceph mds.e for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:23.304 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:23 smithi156 podman[155731]: 2023-11-16T23:58:23.607 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:23 smithi156 ceph-mds[155772]: -- 172.21.15.156:0/2269574030 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5556d5a01520 con 0x5556d59d6800 2023-11-16T23:58:23.607 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: starting mds.e at 2023-11-16T23:58:23.607 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:23 smithi156 bash[155731]: ef9fcd63aa977df5cd79e2ee780bd731b2c5cb2d643c0381dc5e79e17f25a293 2023-11-16T23:58:23.607 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 16 23:58:23 smithi156 systemd[1]: Started Ceph mds.e for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:23.932 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:24.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.150+0000 7f7cb0bff700 1 -- 172.21.15.23:0/388994347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 msgr2=0x7f7cac1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:24.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.150+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/388994347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac1006a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009f90 tx=0x7f7c9c00b330 comp rx=0 tx=0).stop 2023-11-16T23:58:24.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 -- 172.21.15.23:0/388994347 shutdown_connections 2023-11-16T23:58:24.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/388994347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101e10 0x7f7cac1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/388994347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac1006a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/388994347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101480 0x7f7cac1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 -- 172.21.15.23:0/388994347 >> 172.21.15.23:0/388994347 conn(0x7f7cac0fb870 msgr2=0x7f7cac0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 -- 172.21.15.23:0/388994347 shutdown_connections 2023-11-16T23:58:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.151+0000 7f7cb0bff700 1 -- 172.21.15.23:0/388994347 wait complete. 2023-11-16T23:58:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 Processor -- start 2023-11-16T23:58:24.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 -- start start 2023-11-16T23:58:24.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 0x7f7cac072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 0x7f7cac073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac073bc0 con 0x7f7cac101e10 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac073d30 con 0x7f7cac101480 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7cb0bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac19ff80 con 0x7f7cac100290 2023-11-16T23:58:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7caad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 0x7f7cac073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7caa59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7caa59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:43778/0 (socket says 172.21.15.23:43778) 2023-11-16T23:58:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7caa59c700 1 -- 172.21.15.23:0/3356305909 learned_addr learned my addr 172.21.15.23:0/3356305909 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.152+0000 7f7ca3fff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 0x7f7cac072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 -- 172.21.15.23:0/3356305909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 msgr2=0x7f7cac0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 -- 172.21.15.23:0/3356305909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 msgr2=0x7f7cac072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 0x7f7cac072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 -- 172.21.15.23:0/3356305909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c940095f0 con 0x7f7cac101e10 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7ca3fff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 0x7f7cac072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:58:24.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caa59c700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:24.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7caad9d700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 0x7f7cac073420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400e970 tx=0x7f7ca400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:24.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca400cbd0 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c009c40 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ca400ee60 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4004ea0 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.153+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac10bb90 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.154+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac067a60 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.156+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7ca400cd30 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.159+0000 7f7ca37fe700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 0x7f7c980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.159+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7ca4014070 con 0x7f7cac101e10 2023-11-16T23:58:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.159+0000 7f7caa59c700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 0x7f7c980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.159+0000 7f7caa59c700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 0x7f7c980a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0079e0 tx=0x7f7c9c0096c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.163+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7ca4099a10 con 0x7f7cac101e10 2023-11-16T23:58:24.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.418+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cac107210 con 0x7f7cac101e10 2023-11-16T23:58:24.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.419+0000 7f7ca37fe700 1 -- 172.21.15.23:0/3356305909 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4294 (secure 0 0 0) 0x7f7ca40995a0 con 0x7f7cac101e10 2023-11-16T23:58:24.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:24.453 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-11-16T23:58:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 msgr2=0x7f7c980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 0x7f7c980a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0079e0 tx=0x7f7c9c0096c0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 msgr2=0x7f7cac073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 0x7f7cac073420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400e970 tx=0x7f7ca400ec80 comp rx=0 tx=0).stop 2023-11-16T23:58:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 shutdown_connections 2023-11-16T23:58:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cac101e10 0x7f7cac073420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7c980a1150 0x7f7c980a3600 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.421+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cac100290 0x7f7cac0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.422+0000 7f7cb0bff700 1 --2- 172.21.15.23:0/3356305909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cac101480 0x7f7cac072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.422+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 >> 172.21.15.23:0/3356305909 conn(0x7f7cac0fb870 msgr2=0x7f7cac0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.422+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 shutdown_connections 2023-11-16T23:58:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:24.422+0000 7f7cb0bff700 1 -- 172.21.15.23:0/3356305909 wait complete. 2023-11-16T23:58:24.573 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6834/3543343885,v1:172.21.15.156:6835/3543343885] up:boot 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: fsmap 5 up:standby 2023-11-16T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6834/3543343885,v1:172.21.15.156:6835/3543343885] up:boot 2023-11-16T23:58:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: fsmap 5 up:standby 2023-11-16T23:58:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T23:58:24.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6834/3543343885,v1:172.21.15.156:6835/3543343885] up:boot 2023-11-16T23:58:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: fsmap 5 up:standby 2023-11-16T23:58:24.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T23:58:25.331 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 systemd[1]: Starting Ceph mds.f for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:25.598 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:25 smithi173 ceph-mon[131534]: Deploying daemon mds.f on smithi173 2023-11-16T23:58:25.598 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:25 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3356305909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:25.598 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 podman[157000]: 2023-11-16T23:58:25.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:25 smithi156 ceph-mon[131105]: Deploying daemon mds.f on smithi173 2023-11-16T23:58:25.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:25 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3356305909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:25.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:25 smithi023 ceph-mon[97824]: Deploying daemon mds.f on smithi173 2023-11-16T23:58:25.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:25 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3356305909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:25.856 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 bash[157000]: 205b78625cbda14e859974b17244e257a60c9a6e47a1ef4384490125c78e9999 2023-11-16T23:58:25.856 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 ceph-mds[157040]: -- 172.21.15.173:0/3632018731 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a221e9b520 con 0x55a221e70800 2023-11-16T23:58:25.857 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: starting mds.f at 2023-11-16T23:58:25.857 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 16 23:58:25 smithi173 systemd[1]: Started Ceph mds.f for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:26.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:26.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:26.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:26.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:26.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:26.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.789+0000 7f78f95ff700 1 -- 172.21.15.23:0/2533248897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f41014a0 msgr2=0x7f78f41018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.789+0000 7f78f95ff700 1 --2- 172.21.15.23:0/2533248897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f41014a0 0x7f78f41018f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78e80098f0 tx=0x7f78e8009c00 comp rx=0 tx=0).stop 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 -- 172.21.15.23:0/2533248897 shutdown_connections 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 --2- 172.21.15.23:0/2533248897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f4101e30 0x7f78f41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 --2- 172.21.15.23:0/2533248897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f41006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 --2- 172.21.15.23:0/2533248897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f41014a0 0x7f78f41018f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 -- 172.21.15.23:0/2533248897 >> 172.21.15.23:0/2533248897 conn(0x7f78f40fb890 msgr2=0x7f78f40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 -- 172.21.15.23:0/2533248897 shutdown_connections 2023-11-16T23:58:26.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.790+0000 7f78f95ff700 1 -- 172.21.15.23:0/2533248897 wait complete. 2023-11-16T23:58:26.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 Processor -- start 2023-11-16T23:58:26.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 -- start start 2023-11-16T23:58:26.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f41014a0 0x7f78f4196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 0x7f78f419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f419db50 con 0x7f78f41002b0 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f419dc90 con 0x7f78f41014a0 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f95ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f419de20 con 0x7f78f4101e30 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52386/0 (socket says 172.21.15.23:52386) 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f2ffd700 1 -- 172.21.15.23:0/1612935415 learned_addr learned my addr 172.21.15.23:0/1612935415 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.791+0000 7f78f37fe700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 0x7f78f419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f27fc700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f41014a0 0x7f78f4196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:26.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 -- 172.21.15.23:0/1612935415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 msgr2=0x7f78f419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 0x7f78f419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 -- 172.21.15.23:0/1612935415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f41014a0 msgr2=0x7f78f4196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f41014a0 0x7f78f4196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 -- 172.21.15.23:0/1612935415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e8009520 con 0x7f78f41002b0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f37fe700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 0x7f78f419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f2ffd700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f78e400e9f0 tx=0x7f78e400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4004220 con 0x7f78f41002b0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78e4004380 con 0x7f78f41002b0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f410bc10 con 0x7f78f41002b0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.792+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4003be0 con 0x7f78f41002b0 2023-11-16T23:58:26.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.793+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f410ba80 con 0x7f78f41002b0 2023-11-16T23:58:26.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.794+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f41078e0 con 0x7f78f41002b0 2023-11-16T23:58:26.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.796+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f78e4003d40 con 0x7f78f41002b0 2023-11-16T23:58:26.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.797+0000 7f78dbfff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 0x7f78dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:26.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.797+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f78e4014070 con 0x7f78f41002b0 2023-11-16T23:58:26.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.797+0000 7f78f27fc700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 0x7f78dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:26.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.798+0000 7f78f27fc700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 0x7f78dc0a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f78e8005e90 tx=0x7f78e8019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:26.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:26.801+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f78e4096100 con 0x7f78f41002b0 2023-11-16T23:58:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.059+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f402cfe0 con 0x7f78f41002b0 2023-11-16T23:58:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.059+0000 7f78dbfff700 1 -- 172.21.15.23:0/1612935415 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f78e4099720 con 0x7f78f41002b0 2023-11-16T23:58:27.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:27.060 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 msgr2=0x7f78dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 0x7f78dc0a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f78e8005e90 tx=0x7f78e8019040 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 msgr2=0x7f78f4195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f78e400e9f0 tx=0x7f78e400ed00 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 shutdown_connections 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78dc0a1150 0x7f78dc0a3600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78f4101e30 0x7f78f419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78f41002b0 0x7f78f4195fd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 --2- 172.21.15.23:0/1612935415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78f41014a0 0x7f78f4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 >> 172.21.15.23:0/1612935415 conn(0x7f78f40fb890 msgr2=0x7f78f4104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 shutdown_connections 2023-11-16T23:58:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:27.062+0000 7f78f95ff700 1 -- 172.21.15.23:0/1612935415 wait complete. 2023-11-16T23:58:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2023-11-16T23:58:27.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:27 smithi156 ceph-mon[131105]: Deploying daemon mds.c on smithi173 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:27 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:boot 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:27 smithi156 ceph-mon[131105]: fsmap 6 up:standby 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:27 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:27 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1612935415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mon[131534]: Deploying daemon mds.c on smithi173 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:boot 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mon[131534]: fsmap 6 up:standby 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T23:58:27.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1612935415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:27.609 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 systemd[1]: Starting Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:27.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:27 smithi023 ceph-mon[97824]: Deploying daemon mds.c on smithi173 2023-11-16T23:58:27.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:27 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:boot 2023-11-16T23:58:27.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:27 smithi023 ceph-mon[97824]: fsmap 6 up:standby 2023-11-16T23:58:27.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:27 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T23:58:27.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:27 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1612935415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:27.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 podman[157340]: 2023-11-16T23:58:27.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-mds[157380]: -- 172.21.15.173:0/637070188 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f29a6f520 con 0x561f29a44800 2023-11-16T23:58:27.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: starting mds.c at 2023-11-16T23:58:28.349 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 bash[157340]: d648d773423a1589b715d20590ae0b956ff9cc652de5a96ecf4d3dd4a22f12f7 2023-11-16T23:58:28.349 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 16 23:58:27 smithi173 systemd[1]: Started Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:28.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:28.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:28.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:28.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:28.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:28.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:29.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 -- 172.21.15.23:0/3119071151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 msgr2=0x7ff4301006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 --2- 172.21.15.23:0/3119071151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff4301006a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff424009990 tx=0x7ff424009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 -- 172.21.15.23:0/3119071151 shutdown_connections 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 --2- 172.21.15.23:0/3119071151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 0x7ff4301086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 --2- 172.21.15.23:0/3119071151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff4301006a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 --2- 172.21.15.23:0/3119071151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff4301018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 -- 172.21.15.23:0/3119071151 >> 172.21.15.23:0/3119071151 conn(0x7ff4300fb870 msgr2=0x7ff4300fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.299+0000 7ff436bff700 1 -- 172.21.15.23:0/3119071151 shutdown_connections 2023-11-16T23:58:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.300+0000 7ff436bff700 1 -- 172.21.15.23:0/3119071151 wait complete. 2023-11-16T23:58:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.300+0000 7ff436bff700 1 Processor -- start 2023-11-16T23:58:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 -- start start 2023-11-16T23:58:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff430196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff430196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 0x7ff43019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43019db90 con 0x7ff430100290 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43019dcd0 con 0x7ff430101480 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff436bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43019de60 con 0x7ff430101e10 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff43499b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff430196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff42ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff430196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff42ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff430196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:59954/0 (socket says 172.21.15.23:59954) 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff42ffff700 1 -- 172.21.15.23:0/1743677697 learned_addr learned my addr 172.21.15.23:0/1743677697 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff43519c700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 0x7ff43019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff43499b700 1 -- 172.21.15.23:0/1743677697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 msgr2=0x7ff43019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.301+0000 7ff43499b700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 0x7ff43019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff43499b700 1 -- 172.21.15.23:0/1743677697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 msgr2=0x7ff430196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff43499b700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff430196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff43499b700 1 -- 172.21.15.23:0/1743677697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240095f0 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff43499b700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff430196010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4240098e0 tx=0x7ff424005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42401d070 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff424022470 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff424017650 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43010bc40 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.302+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43010bab0 con 0x7ff430100290 2023-11-16T23:58:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.303+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff430107870 con 0x7ff430100290 2023-11-16T23:58:29.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.305+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff424022870 con 0x7ff430100290 2023-11-16T23:58:29.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.308+0000 7ff42dffb700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 0x7ff4200a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:29.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.308+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4240cda80 con 0x7ff430100290 2023-11-16T23:58:29.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.308+0000 7ff42ffff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 0x7ff4200a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:29.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.308+0000 7ff42ffff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 0x7ff4200a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009d20 tx=0x7ff41c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:29.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.312+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff424097660 con 0x7ff430100290 2023-11-16T23:58:29.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:29 smithi023 ceph-mon[97824]: Deploying daemon mds.h on smithi156 2023-11-16T23:58:29.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:29 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6836/1936617634,v1:172.21.15.173:6837/1936617634] up:boot 2023-11-16T23:58:29.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:29 smithi023 ceph-mon[97824]: fsmap 7 up:standby 2023-11-16T23:58:29.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T23:58:29.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:29 smithi023 ceph-mon[97824]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.568+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff43002cfe0 con 0x7ff430100290 2023-11-16T23:58:29.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.569+0000 7ff42dffb700 1 -- 172.21.15.23:0/1743677697 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7ff42409ac80 con 0x7ff430100290 2023-11-16T23:58:29.570 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:29.570 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T23:58:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 msgr2=0x7ff4200a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 0x7ff4200a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009d20 tx=0x7ff41c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 msgr2=0x7ff430196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff430196010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4240098e0 tx=0x7ff424005740 comp rx=0 tx=0).stop 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 shutdown_connections 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4200a11d0 0x7ff4200a3680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff430101e10 0x7ff43019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff430100290 0x7ff430196010 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 --2- 172.21.15.23:0/1743677697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff430101480 0x7ff430196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 >> 172.21.15.23:0/1743677697 conn(0x7ff4300fb870 msgr2=0x7ff4301049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 shutdown_connections 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:29.571+0000 7ff436bff700 1 -- 172.21.15.23:0/1743677697 wait complete. 2023-11-16T23:58:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2023-11-16T23:58:29.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:29 smithi173 ceph-mon[131534]: Deploying daemon mds.h on smithi156 2023-11-16T23:58:29.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:29 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6836/1936617634,v1:172.21.15.173:6837/1936617634] up:boot 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:29 smithi173 ceph-mon[131534]: fsmap 7 up:standby 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:29 smithi173 ceph-mon[131534]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:29 smithi156 systemd[1]: Starting Ceph mds.h for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:29 smithi156 ceph-mon[131105]: Deploying daemon mds.h on smithi156 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:29 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6836/1936617634,v1:172.21.15.173:6837/1936617634] up:boot 2023-11-16T23:58:29.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:29 smithi156 ceph-mon[131105]: fsmap 7 up:standby 2023-11-16T23:58:29.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T23:58:29.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:29 smithi156 ceph-mon[131105]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:30.078 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:29 smithi156 podman[156075]: 2023-11-16T23:58:30.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mds[156115]: -- 172.21.15.156:0/1860521124 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dd04df1520 con 0x55dd04dc6800 2023-11-16T23:58:30.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: starting mds.h at 2023-11-16T23:58:30.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:30 smithi156 bash[156075]: 24d30aef9aef3899e2462f67b8eb85a3820c9621f01409b48644b06e3ee64a99 2023-11-16T23:58:30.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 16 23:58:30 smithi156 systemd[1]: Started Ceph mds.h for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1743677697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:30.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:30 smithi023 ceph-mon[97824]: Deploying daemon mds.i on smithi173 2023-11-16T23:58:30.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1743677697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:30.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:30 smithi173 ceph-mon[131534]: Deploying daemon mds.i on smithi173 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1743677697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:30.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:30 smithi156 ceph-mon[131105]: Deploying daemon mds.i on smithi173 2023-11-16T23:58:31.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:31.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:31 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6836/180108175,v1:172.21.15.156:6837/180108175] up:boot 2023-11-16T23:58:31.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:31 smithi173 ceph-mon[131534]: fsmap 8 up:standby 2023-11-16T23:58:31.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T23:58:31.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:31 smithi173 ceph-mon[131534]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:31.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 -- 172.21.15.23:0/1710155706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 msgr2=0x7fcfcc1018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/1710155706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1018f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009990 tx=0x7fcfbc009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 -- 172.21.15.23:0/1710155706 shutdown_connections 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/1710155706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc101e30 0x7fcfcc1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/1710155706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc1002b0 0x7fcfcc1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/1710155706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1018f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 -- 172.21.15.23:0/1710155706 >> 172.21.15.23:0/1710155706 conn(0x7fcfcc0fb890 msgr2=0x7fcfcc0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 -- 172.21.15.23:0/1710155706 shutdown_connections 2023-11-16T23:58:31.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.809+0000 7fcfd41ff700 1 -- 172.21.15.23:0/1710155706 wait complete. 2023-11-16T23:58:31.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 Processor -- start 2023-11-16T23:58:31.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 -- start start 2023-11-16T23:58:31.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc1002b0 0x7fcfcc198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:31.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:31.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc101e30 0x7fcfcc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19fd10 con 0x7fcfcc1014a0 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19fe50 con 0x7fcfcc1002b0 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.810+0000 7fcfd41ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19ffe0 con 0x7fcfcc101e30 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34338/0 (socket says 172.21.15.23:34338) 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 -- 172.21.15.23:0/927661738 learned_addr learned my addr 172.21.15.23:0/927661738 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd279c700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc101e30 0x7fcfcc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd1f9b700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc1002b0 0x7fcfcc198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 -- 172.21.15.23:0/927661738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc101e30 msgr2=0x7fcfcc19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc101e30 0x7fcfcc19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 -- 172.21.15.23:0/927661738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc1002b0 msgr2=0x7fcfcc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc1002b0 0x7fcfcc198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 -- 172.21.15.23:0/927661738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc0095f0 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.811+0000 7fcfd179a700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b620 tx=0x7fcfbc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.812+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc01d070 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.812+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfbc022470 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.812+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc10bc60 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.812+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc017650 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.812+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc10bad0 con 0x7fcfcc1014a0 2023-11-16T23:58:31.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.813+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc1928f0 con 0x7fcfcc1014a0 2023-11-16T23:58:31.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.815+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcfbc0225e0 con 0x7fcfcc1014a0 2023-11-16T23:58:31.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.816+0000 7fcfc2ffd700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 0x7fcfb80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:31.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.816+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcfbc0cda10 con 0x7fcfcc1014a0 2023-11-16T23:58:31.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.816+0000 7fcfd1f9b700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 0x7fcfb80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:31.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.817+0000 7fcfd1f9b700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 0x7fcfb80a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc101300 tx=0x7fcfc8007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:31.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:31.820+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcfbc0975f0 con 0x7fcfcc1014a0 2023-11-16T23:58:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:31 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6836/180108175,v1:172.21.15.156:6837/180108175] up:boot 2023-11-16T23:58:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:31 smithi023 ceph-mon[97824]: fsmap 8 up:standby 2023-11-16T23:58:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T23:58:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:31 smithi023 ceph-mon[97824]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:32.073 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:31 smithi173 systemd[1]: Starting Ceph mds.i for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.073+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfcc02cfe0 con 0x7fcfcc1014a0 2023-11-16T23:58:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.074+0000 7fcfc2ffd700 1 -- 172.21.15.23:0/927661738 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7fcfbc09ac10 con 0x7fcfcc1014a0 2023-11-16T23:58:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:32.074 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 msgr2=0x7fcfb80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 0x7fcfb80a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc101300 tx=0x7fcfc8007400 comp rx=0 tx=0).stop 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 msgr2=0x7fcfcc1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b620 tx=0x7fcfbc005740 comp rx=0 tx=0).stop 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 shutdown_connections 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcfb80a11d0 0x7fcfb80a3680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfcc101e30 0x7fcfcc19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfcc1002b0 0x7fcfcc198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 --2- 172.21.15.23:0/927661738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfcc1014a0 0x7fcfcc1986d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 >> 172.21.15.23:0/927661738 conn(0x7fcfcc0fb890 msgr2=0x7fcfcc106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 shutdown_connections 2023-11-16T23:58:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:32.076+0000 7fcfd41ff700 1 -- 172.21.15.23:0/927661738 wait complete. 2023-11-16T23:58:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 9 2023-11-16T23:58:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:31 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6836/180108175,v1:172.21.15.156:6837/180108175] up:boot 2023-11-16T23:58:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:31 smithi156 ceph-mon[131105]: fsmap 8 up:standby 2023-11-16T23:58:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T23:58:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:31 smithi156 ceph-mon[131105]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:32.330 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:32 smithi173 podman[157682]: 2023-11-16T23:58:32.330 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mds[157722]: -- 172.21.15.173:0/3761232820 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b143d41520 con 0x55b143d16c00 2023-11-16T23:58:32.330 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: starting mds.i at 2023-11-16T23:58:32.606 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:32 smithi173 bash[157682]: 41bc786cfd480bacb5feedddc904dd8bfbb3c4feaeffc783f6fc02046f570035 2023-11-16T23:58:32.606 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 16 23:58:32 smithi173 systemd[1]: Started Ceph mds.i for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/927661738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:33.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:33.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/927661738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:33.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/927661738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:33.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:33.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:33.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:33.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:33.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:33 smithi023 ceph-mon[97824]: Deploying daemon mds.g on smithi023 2023-11-16T23:58:33.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:33 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:boot 2023-11-16T23:58:33.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:33 smithi023 ceph-mon[97824]: fsmap 9 up:standby 2023-11-16T23:58:33.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:33 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T23:58:33.948 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:33 smithi023 ceph-mon[97824]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:34.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:33 smithi156 ceph-mon[131105]: Deploying daemon mds.g on smithi023 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:33 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:boot 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:33 smithi156 ceph-mon[131105]: fsmap 9 up:standby 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:33 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:33 smithi156 ceph-mon[131105]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:33 smithi173 ceph-mon[131534]: Deploying daemon mds.g on smithi023 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:33 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:boot 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:33 smithi173 ceph-mon[131534]: fsmap 9 up:standby 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:33 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T23:58:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:33 smithi173 ceph-mon[131534]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:34.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 -- 172.21.15.23:0/1589820772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 msgr2=0x7fbd10072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:34.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 --2- 172.21.15.23:0/1589820772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10072f80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009990 tx=0x7fbd00009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 -- 172.21.15.23:0/1589820772 shutdown_connections 2023-11-16T23:58:34.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 --2- 172.21.15.23:0/1589820772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10108150 0x7fbd1010a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 --2- 172.21.15.23:0/1589820772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10072f80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 --2- 172.21.15.23:0/1589820772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10073550 0x7fbd100739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 -- 172.21.15.23:0/1589820772 >> 172.21.15.23:0/1589820772 conn(0x7fbd100fb870 msgr2=0x7fbd100fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:34.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 -- 172.21.15.23:0/1589820772 shutdown_connections 2023-11-16T23:58:34.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.327+0000 7fbd159ff700 1 -- 172.21.15.23:0/1589820772 wait complete. 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 Processor -- start 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 -- start start 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 0x7fbd101981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1019fd70 con 0x7fbd10108150 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1019feb0 con 0x7fbd10074b30 2023-11-16T23:58:34.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.329+0000 7fbd159ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd101a0040 con 0x7fbd10073550 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 0x7fbd101981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34350/0 (socket says 172.21.15.23:34350) 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 -- 172.21.15.23:0/4139111969 learned_addr learned my addr 172.21.15.23:0/4139111969 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 -- 172.21.15.23:0/4139111969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 msgr2=0x7fbd101981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 0x7fbd101981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 -- 172.21.15.23:0/4139111969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 msgr2=0x7fbd10198730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0e7fc700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.330+0000 7fbd0f7fe700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd0f7fe700 1 -- 172.21.15.23:0/4139111969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd000095f0 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd0effd700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 0x7fbd101981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd0f7fe700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbd08007ee0 tx=0x7fbd0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08017070 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd0800f040 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1010de50 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08005740 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.331+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1010dcc0 con 0x7fbd10108150 2023-11-16T23:58:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.332+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10106800 con 0x7fbd10108150 2023-11-16T23:58:34.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.335+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbd0800dc60 con 0x7fbd10108150 2023-11-16T23:58:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.336+0000 7fbd149fd700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 0x7fbcf80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.336+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbd080cbc40 con 0x7fbd10108150 2023-11-16T23:58:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.336+0000 7fbd0effd700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 0x7fbcf80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.337+0000 7fbd0effd700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 0x7fbcf80a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000b5c0 tx=0x7fbd00005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:34.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.341+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbd08095830 con 0x7fbd10108150 2023-11-16T23:58:34.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.603+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd1002cfe0 con 0x7fbd10108150 2023-11-16T23:58:34.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.603+0000 7fbd149fd700 1 -- 172.21.15.23:0/4139111969 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7fbd08098e50 con 0x7fbd10108150 2023-11-16T23:58:34.626 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:34.626 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 msgr2=0x7fbcf80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 0x7fbcf80a3600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000b5c0 tx=0x7fbd00005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 msgr2=0x7fbd1019f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbd08007ee0 tx=0x7fbd0800d3b0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 shutdown_connections 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd10108150 0x7fbd1019f7a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbcf80a1150 0x7fbcf80a3600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd10073550 0x7fbd101981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 --2- 172.21.15.23:0/4139111969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd10074b30 0x7fbd10198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 >> 172.21.15.23:0/4139111969 conn(0x7fbd100fb870 msgr2=0x7fbd100fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 shutdown_connections 2023-11-16T23:58:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:34.606+0000 7fbd159ff700 1 -- 172.21.15.23:0/4139111969 wait complete. 2023-11-16T23:58:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2023-11-16T23:58:34.873 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:34 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4139111969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:34.873 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:34 smithi023 systemd[1]: Starting Ceph mds.g for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:35.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:34 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4139111969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:35.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:34 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4139111969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:35.619 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:35 smithi023 podman[131414]: 2023-11-16T23:58:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:35 smithi023 ceph-mon[97824]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:35 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:35 smithi023 bash[131414]: 5bbb35e14b84a6413b2499fa18febf57eeb3c88a0b40ebf69e921198c8d5d0e1 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:35 smithi023 ceph-mds[131454]: -- 172.21.15.23:0/2605207333 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56079260f520 con 0x5607925e7000 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-g[131430]: starting mds.g at 2023-11-16T23:58:36.018 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 16 23:58:35 smithi023 systemd[1]: Started Ceph mds.g for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:36.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:35 smithi173 ceph-mon[131534]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:36.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:35 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:35 smithi156 ceph-mon[131105]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:36.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:35 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:36.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.842+0000 7f3f21bff700 1 -- 172.21.15.23:0/709899543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c1036f0 msgr2=0x7f3f1c105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.842+0000 7f3f21bff700 1 --2- 172.21.15.23:0/709899543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c105ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009990 tx=0x7f3f10009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 -- 172.21.15.23:0/709899543 shutdown_connections 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 --2- 172.21.15.23:0/709899543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c106010 0x7f3f1c1083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 --2- 172.21.15.23:0/709899543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 --2- 172.21.15.23:0/709899543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c105ad0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 -- 172.21.15.23:0/709899543 >> 172.21.15.23:0/709899543 conn(0x7f3f1c0fa750 msgr2=0x7f3f1c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 -- 172.21.15.23:0/709899543 shutdown_connections 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.843+0000 7f3f21bff700 1 -- 172.21.15.23:0/709899543 wait complete. 2023-11-16T23:58:36.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.844+0000 7f3f21bff700 1 Processor -- start 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.844+0000 7f3f21bff700 1 -- start start 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c106010 0x7f3f1c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c19db40 con 0x7f3f1c1036f0 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c19dc80 con 0x7f3f1c100dd0 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f21bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c19de10 con 0x7f3f1c106010 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f1b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f1affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f1b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:54906/0 (socket says 172.21.15.23:54906) 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f1b7fe700 1 -- 172.21.15.23:0/2917503917 learned_addr learned my addr 172.21.15.23:0/2917503917 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:36.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.845+0000 7f3f1bfff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c106010 0x7f3f1c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:36.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 -- 172.21.15.23:0/2917503917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c106010 msgr2=0x7f3f1c19d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c106010 0x7f3f1c19d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 -- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c1036f0 msgr2=0x7f3f1c196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 -- 172.21.15.23:0/2917503917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100095f0 con 0x7f3f1c100dd0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f1b7fe700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e970 tx=0x7f3f0c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c009790 con 0x7f3f1c100dd0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c10bc50 con 0x7f3f1c100dd0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.846+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f0c00c7e0 con 0x7f3f1c100dd0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.847+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c0054e0 con 0x7f3f1c100dd0 2023-11-16T23:58:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.847+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c10bac0 con 0x7f3f1c100dd0 2023-11-16T23:58:36.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.848+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c19e5b0 con 0x7f3f1c100dd0 2023-11-16T23:58:36.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.850+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3f0c00c950 con 0x7f3f1c100dd0 2023-11-16T23:58:36.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.851+0000 7f3f18ff9700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 0x7f3f040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.851+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f0c014070 con 0x7f3f1c100dd0 2023-11-16T23:58:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.851+0000 7f3f1affd700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 0x7f3f040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.852+0000 7f3f1affd700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 0x7f3f040a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000b2e0 tx=0x7f3f10005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:36.855+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3f0c096270 con 0x7f3f1c100dd0 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.23:6838/3704661789,v1:172.21.15.23:6839/3704661789] up:boot 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: fsmap 10 up:standby 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:36.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:36 smithi023 ceph-mon[97824]: Deploying daemon mds.a on smithi023 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.23:6838/3704661789,v1:172.21.15.23:6839/3704661789] up:boot 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: fsmap 10 up:standby 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:37.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:36 smithi173 ceph-mon[131534]: Deploying daemon mds.a on smithi023 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.23:6838/3704661789,v1:172.21.15.23:6839/3704661789] up:boot 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: fsmap 10 up:standby 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:37.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:37.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:36 smithi156 ceph-mon[131105]: Deploying daemon mds.a on smithi023 2023-11-16T23:58:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.123+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1c19eb60 con 0x7f3f1c100dd0 2023-11-16T23:58:37.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.127+0000 7f3f18ff9700 1 -- 172.21.15.23:0/2917503917 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7f3f0c099890 con 0x7f3f1c100dd0 2023-11-16T23:58:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:37.128 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 msgr2=0x7f3f040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 0x7f3f040a3600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000b2e0 tx=0x7f3f10005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 msgr2=0x7f3f1c195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e970 tx=0x7f3f0c00ec80 comp rx=0 tx=0).stop 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 shutdown_connections 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3f040a1150 0x7f3f040a3600 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f1c106010 0x7f3f1c19d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f1c100dd0 0x7f3f1c195fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 --2- 172.21.15.23:0/2917503917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f1c1036f0 0x7f3f1c196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 >> 172.21.15.23:0/2917503917 conn(0x7f3f1c0fa750 msgr2=0x7f3f1c102230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 shutdown_connections 2023-11-16T23:58:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:37.129+0000 7f3f21bff700 1 -- 172.21.15.23:0/2917503917 wait complete. 2023-11-16T23:58:37.133 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2023-11-16T23:58:38.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:37 smithi023 ceph-mon[97824]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:37 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2917503917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:38.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:37 smithi173 ceph-mon[131534]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:38.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:37 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2917503917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:38.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:37 smithi156 ceph-mon[131105]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:38.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:37 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2917503917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:38.518 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 systemd[1]: Starting Ceph mds.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:38.809 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 podman[131802]: 2023-11-16T23:58:39.151 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:39.216 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 bash[131802]: 60d1e85599ab63c04daee972858ece26979f27cf6d886ba59523b50d951b6978 2023-11-16T23:58:39.216 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 ceph-mds[131842]: -- 172.21.15.23:0/485404430 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ea8e3f520 con 0x557ea8e16c00 2023-11-16T23:58:39.216 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-a[131818]: starting mds.a at 2023-11-16T23:58:39.216 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 16 23:58:38 smithi023 systemd[1]: Started Ceph mds.a for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2786137094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 msgr2=0x7f6ee81086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2786137094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee81086f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009f90 tx=0x7f6ee000b330 comp rx=0 tx=0).stop 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2786137094 shutdown_connections 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2786137094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee81086f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2786137094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81002b0 0x7f6ee81006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2786137094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81014a0 0x7f6ee81018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2786137094 >> 172.21.15.23:0/2786137094 conn(0x7f6ee80fb890 msgr2=0x7f6ee80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2786137094 shutdown_connections 2023-11-16T23:58:39.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.360+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2786137094 wait complete. 2023-11-16T23:58:39.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.361+0000 7f6ef01ff700 1 Processor -- start 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.361+0000 7f6ef01ff700 1 -- start start 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 0x7f6ee8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8073bc0 con 0x7f6ee81002b0 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8073d30 con 0x7f6ee81014a0 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6ef01ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee819ff00 con 0x7f6ee8101e30 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34426/0 (socket says 172.21.15.23:34426) 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 -- 172.21.15.23:0/2844530235 learned_addr learned my addr 172.21.15.23:0/2844530235 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eed79a700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 0x7f6ee8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eee79c700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:39.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 -- 172.21.15.23:0/2844530235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 msgr2=0x7f6ee8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 -- 172.21.15.23:0/2844530235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 msgr2=0x7f6ee8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 0x7f6ee8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eedf9b700 1 -- 172.21.15.23:0/2844530235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee40095f0 con 0x7f6ee81002b0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eed79a700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 0x7f6ee8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.362+0000 7f6eee79c700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6eedf9b700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400ebf0 tx=0x7f6ee400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4009a10 con 0x7f6ee81002b0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ee400cc90 con 0x7f6ee81002b0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed8009520 con 0x7f6ee81002b0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4018750 con 0x7f6ee81002b0 2023-11-16T23:58:39.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.363+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0009c40 con 0x7f6ee81002b0 2023-11-16T23:58:39.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.364+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee8107800 con 0x7f6ee81002b0 2023-11-16T23:58:39.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.366+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6ee40188b0 con 0x7f6ee81002b0 2023-11-16T23:58:39.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.367+0000 7f6edeffd700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 0x7f6ed40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:39.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.368+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ee4014070 con 0x7f6ee81002b0 2023-11-16T23:58:39.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.368+0000 7f6eed79a700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 0x7f6ed40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:39.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.368+0000 7f6eed79a700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 0x7f6ed40a3680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009fd0 tx=0x7f6ed8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:39.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.372+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6ee40965d0 con 0x7f6ee81002b0 2023-11-16T23:58:39.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.630+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee802cbc0 con 0x7f6ee81002b0 2023-11-16T23:58:39.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.631+0000 7f6edeffd700 1 -- 172.21.15.23:0/2844530235 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7f6ee4099bf0 con 0x7f6ee81002b0 2023-11-16T23:58:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:39.632 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T23:58:39.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 msgr2=0x7f6ed40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:39.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 0x7f6ed40a3680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009fd0 tx=0x7f6ed8009260 comp rx=0 tx=0).stop 2023-11-16T23:58:39.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 msgr2=0x7f6ee80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:39.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400ebf0 tx=0x7f6ee400ef00 comp rx=0 tx=0).stop 2023-11-16T23:58:39.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 shutdown_connections 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6ed40a11d0 0x7f6ed40a3680 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.633+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ee8101e30 0x7f6ee8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.634+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee81002b0 0x7f6ee80748a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.634+0000 7f6ef01ff700 1 --2- 172.21.15.23:0/2844530235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ee81014a0 0x7f6ee8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.634+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 >> 172.21.15.23:0/2844530235 conn(0x7f6ee80fb890 msgr2=0x7f6ee8104940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.634+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 shutdown_connections 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:39.634+0000 7f6ef01ff700 1 -- 172.21.15.23:0/2844530235 wait complete. 2023-11-16T23:58:39.636 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: Deploying daemon mds.b on smithi156 2023-11-16T23:58:40.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:40.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2844530235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:40.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.23:6840/3035227424,v1:172.21.15.23:6841/3035227424] up:boot 2023-11-16T23:58:40.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: fsmap 11 up:standby 2023-11-16T23:58:40.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:39 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:40.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: Deploying daemon mds.b on smithi156 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2844530235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.23:6840/3035227424,v1:172.21.15.23:6841/3035227424] up:boot 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: fsmap 11 up:standby 2023-11-16T23:58:40.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:39 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T23:58:40.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: Deploying daemon mds.b on smithi156 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2844530235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.23:6840/3035227424,v1:172.21.15.23:6841/3035227424] up:boot 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: fsmap 11 up:standby 2023-11-16T23:58:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:39 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T23:58:40.533 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:40 smithi156 systemd[1]: Starting Ceph mds.b for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-16T23:58:41.062 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:40 smithi156 podman[156413]: 2023-11-16T23:58:41.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:41 smithi156 ceph-mds[156453]: -- 172.21.15.156:0/3400056926 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55869c261520 con 0x55869c236800 2023-11-16T23:58:41.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:41 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: starting mds.b at 2023-11-16T23:58:41.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:41 smithi156 bash[156413]: fdd87a7c7ab30c285e7de6ec7fca964112d10ab0e03724e858c4f1fda4879561 2023-11-16T23:58:41.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:41 smithi156 systemd[1]: Started Ceph mds.b for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-16T23:58:41.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:41.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 -- 172.21.15.23:0/4257775576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5894074b30 msgr2=0x7f5894072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 --2- 172.21.15.23:0/4257775576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5894074b30 0x7f5894072f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5890009990 tx=0x7f5890009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 -- 172.21.15.23:0/4257775576 shutdown_connections 2023-11-16T23:58:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 --2- 172.21.15.23:0/4257775576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f589410c860 0x7f589410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 --2- 172.21.15.23:0/4257775576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5894074b30 0x7f5894072f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 --2- 172.21.15.23:0/4257775576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894073550 0x7f58940739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 -- 172.21.15.23:0/4257775576 >> 172.21.15.23:0/4257775576 conn(0x7f58940fb870 msgr2=0x7f58940fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 -- 172.21.15.23:0/4257775576 shutdown_connections 2023-11-16T23:58:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.863+0000 7f589c7ff700 1 -- 172.21.15.23:0/4257775576 wait complete. 2023-11-16T23:58:41.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 Processor -- start 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 -- start start 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5894073550 0x7f589419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a4130 con 0x7f589410c860 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a4270 con 0x7f5894073550 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.865+0000 7f589c7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941a4400 con 0x7f5894074b30 2023-11-16T23:58:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f589ad9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f589a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5894073550 0x7f589419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f589ad9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40676/0 (socket says 172.21.15.23:40676) 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:49268/0 (socket says 172.21.15.23:49268) 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f589ad9c700 1 -- 172.21.15.23:0/3539881839 learned_addr learned my addr 172.21.15.23:0/3539881839 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 -- 172.21.15.23:0/3539881839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5894073550 msgr2=0x7f589419c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5894073550 0x7f589419c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 -- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 msgr2=0x7f58941a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f5899d9a700 1 -- 172.21.15.23:0/3539881839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900095f0 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.866+0000 7f589ad9c700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f5899d9a700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9c0 tx=0x7f588400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884004220 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894112250 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5884004380 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884003be0 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.867+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941120c0 con 0x7f5894074b30 2023-11-16T23:58:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.868+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58941023b0 con 0x7f5894074b30 2023-11-16T23:58:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.870+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5884003d40 con 0x7f5894074b30 2023-11-16T23:58:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.872+0000 7f588b7fe700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 0x7f58800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.872+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5884014070 con 0x7f5894074b30 2023-11-16T23:58:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.872+0000 7f589a59b700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 0x7f58800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.872+0000 7f589a59b700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 0x7f58800a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f589000b2e0 tx=0x7f5890005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:41.876+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f58840962f0 con 0x7f5894074b30 2023-11-16T23:58:42.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.140+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5894061700 con 0x7f5894074b30 2023-11-16T23:58:42.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.141+0000 7f588b7fe700 1 -- 172.21.15.23:0/3539881839 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7f5884099910 con 0x7f5894074b30 2023-11-16T23:58:42.143 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:42.143 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 msgr2=0x7f58800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 0x7f58800a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f589000b2e0 tx=0x7f5890005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 msgr2=0x7f589419caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f588400e9c0 tx=0x7f588400ecd0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 shutdown_connections 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f589410c860 0x7f58941a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5894073550 0x7f589419c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f58800a1150 0x7f58800a3600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 --2- 172.21.15.23:0/3539881839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5894074b30 0x7f589419caf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.143+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 >> 172.21.15.23:0/3539881839 conn(0x7f58940fb870 msgr2=0x7f58940fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.144+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 shutdown_connections 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.144+0000 7f589c7ff700 1 -- 172.21.15.23:0/3539881839 wait complete. 2023-11-16T23:58:42.146 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2023-11-16T23:58:42.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 -- 172.21.15.23:0/3607716862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920073550 msgr2=0x7f29200739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 --2- 172.21.15.23:0/3607716862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920073550 0x7f29200739c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f290c009990 tx=0x7f290c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 -- 172.21.15.23:0/3607716862 shutdown_connections 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 --2- 172.21.15.23:0/3607716862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f292010ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 --2- 172.21.15.23:0/3607716862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920074b30 0x7f2920072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 --2- 172.21.15.23:0/3607716862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920073550 0x7f29200739c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 -- 172.21.15.23:0/3607716862 >> 172.21.15.23:0/3607716862 conn(0x7f29200fb890 msgr2=0x7f29200fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:42.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 -- 172.21.15.23:0/3607716862 shutdown_connections 2023-11-16T23:58:42.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.375+0000 7f2926dff700 1 -- 172.21.15.23:0/3607716862 wait complete. 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 Processor -- start 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 -- start start 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920073550 0x7f292019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 0x7f292019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f29201a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29201a4190 con 0x7f292010c880 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.376+0000 7f2926dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29201a42d0 con 0x7f2920073550 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f2926dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29201a4460 con 0x7f2920074b30 2023-11-16T23:58:42.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f292539c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f29201a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 0x7f292019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f292539c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f29201a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40700/0 (socket says 172.21.15.23:40700) 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f292539c700 1 -- 172.21.15.23:0/1884773651 learned_addr learned my addr 172.21.15.23:0/1884773651 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f2924b9b700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920073550 0x7f292019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 -- 172.21.15.23:0/1884773651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920073550 msgr2=0x7f292019c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920073550 0x7f292019c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 -- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 msgr2=0x7f29201a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f29201a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 -- 172.21.15.23:0/1884773651 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0095f0 con 0x7f2920074b30 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.377+0000 7f291ffff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 0x7f292019cb50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b680 tx=0x7f290c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.378+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c01d070 con 0x7f2920074b30 2023-11-16T23:58:42.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.378+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2920112290 con 0x7f2920074b30 2023-11-16T23:58:42.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.378+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f290c021410 con 0x7f2920074b30 2023-11-16T23:58:42.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.378+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c017650 con 0x7f2920074b30 2023-11-16T23:58:42.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.378+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2920112100 con 0x7f2920074b30 2023-11-16T23:58:42.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.379+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29201023d0 con 0x7f2920074b30 2023-11-16T23:58:42.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.381+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f290c021810 con 0x7f2920074b30 2023-11-16T23:58:42.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.382+0000 7f291dffb700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 0x7f29100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.382+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f290c0cdf60 con 0x7f2920074b30 2023-11-16T23:58:42.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.382+0000 7f2924b9b700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 0x7f29100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.383+0000 7f2924b9b700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 0x7f29100a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2920074f00 tx=0x7f2914009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:42.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.384+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f290c097b40 con 0x7f2920074b30 2023-11-16T23:58:42.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:boot 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: fsmap 12 up:standby 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:42 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:42.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.598+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2920061700 con 0x7f2920074b30 2023-11-16T23:58:42.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.600+0000 7f291dffb700 1 -- 172.21.15.23:0/1884773651 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+18447 (secure 0 0 0) 0x7f290c09b160 con 0x7f2920074b30 2023-11-16T23:58:42.602 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:42.602 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:57:57.694103+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.602+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 msgr2=0x7f29100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 0x7f29100a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2920074f00 tx=0x7f2914009260 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 msgr2=0x7f292019cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 0x7f292019cb50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b680 tx=0x7f290c009580 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 shutdown_connections 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f292010c880 0x7f29201a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2920073550 0x7f292019c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f29100a1150 0x7f29100a3600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 --2- 172.21.15.23:0/1884773651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2920074b30 0x7f292019cb50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 >> 172.21.15.23:0/1884773651 conn(0x7f29200fb890 msgr2=0x7f29200fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 shutdown_connections 2023-11-16T23:58:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.603+0000 7f2926dff700 1 -- 172.21.15.23:0/1884773651 wait complete. 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:boot 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: fsmap 12 up:standby 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:42 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:boot 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: fsmap 12 up:standby 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:42.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:42 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:42.624 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T23:58:42.624 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T23:58:42.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 -- 172.21.15.23:0/2581602105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 msgr2=0x7f5f60072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 --2- 172.21.15.23:0/2581602105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f60072f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009990 tx=0x7f5f5c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 -- 172.21.15.23:0/2581602105 shutdown_connections 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 --2- 172.21.15.23:0/2581602105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f6010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 --2- 172.21.15.23:0/2581602105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f60072f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 --2- 172.21.15.23:0/2581602105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 0x7f5f600739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 -- 172.21.15.23:0/2581602105 >> 172.21.15.23:0/2581602105 conn(0x7f5f600fb8b0 msgr2=0x7f5f600fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.840+0000 7f5f683ff700 1 -- 172.21.15.23:0/2581602105 shutdown_connections 2023-11-16T23:58:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.841+0000 7f5f683ff700 1 -- 172.21.15.23:0/2581602105 wait complete. 2023-11-16T23:58:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.841+0000 7f5f683ff700 1 Processor -- start 2023-11-16T23:58:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 -- start start 2023-11-16T23:58:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 0x7f5f6019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f601a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601a4220 con 0x7f5f60074b30 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601a4360 con 0x7f5f6010c8a0 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f683ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601a44f0 con 0x7f5f60073550 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f6599a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f6599a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40712/0 (socket says 172.21.15.23:40712) 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f6599a700 1 -- 172.21.15.23:0/1609381132 learned_addr learned my addr 172.21.15.23:0/1609381132 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f6699c700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f601a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.842+0000 7f5f6619b700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 0x7f5f6019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 -- 172.21.15.23:0/1609381132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 msgr2=0x7f5f6019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 0x7f5f6019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 -- 172.21.15.23:0/1609381132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 msgr2=0x7f5f601a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f601a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 -- 172.21.15.23:0/1609381132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50009520 con 0x7f5f60074b30 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6699c700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f601a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f6599a700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000ea50 tx=0x7f5f5000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000cbd0 con 0x7f5f60074b30 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f5000cd30 con 0x7f5f60074b30 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c0095f0 con 0x7f5f60074b30 2023-11-16T23:58:42.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.843+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50018680 con 0x7f5f60074b30 2023-11-16T23:58:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.844+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60112290 con 0x7f5f60074b30 2023-11-16T23:58:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.845+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f601023f0 con 0x7f5f60074b30 2023-11-16T23:58:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.846+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5f50010430 con 0x7f5f60074b30 2023-11-16T23:58:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.848+0000 7f5f577fe700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 0x7f5f4c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:42.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.848+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5f50014070 con 0x7f5f60074b30 2023-11-16T23:58:42.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.848+0000 7f5f6619b700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 0x7f5f4c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:42.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.849+0000 7f5f6619b700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 0x7f5f4c0a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00b2e0 tx=0x7f5f5c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:42.852+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5f50096470 con 0x7f5f60074b30 2023-11-16T23:58:43.070 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.068+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f60067b00 con 0x7f5f60074b30 2023-11-16T23:58:43.070 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.069+0000 7f5f577fe700 1 -- 172.21.15.23:0/1609381132 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f50099a90 con 0x7f5f60074b30 2023-11-16T23:58:43.071 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-16T23:58:43.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 msgr2=0x7f5f4c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 0x7f5f4c0a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00b2e0 tx=0x7f5f5c005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 msgr2=0x7f5f6019cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000ea50 tx=0x7f5f5000ed60 comp rx=0 tx=0).stop 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 shutdown_connections 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f6010c8a0 0x7f5f601a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.071+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5f4c0a11d0 0x7f5f4c0a3680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.072+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f60073550 0x7f5f6019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.072+0000 7f5f683ff700 1 --2- 172.21.15.23:0/1609381132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f60074b30 0x7f5f6019cbe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.072+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 >> 172.21.15.23:0/1609381132 conn(0x7f5f600fb8b0 msgr2=0x7f5f600fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.072+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 shutdown_connections 2023-11-16T23:58:43.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.072+0000 7f5f683ff700 1 -- 172.21.15.23:0/1609381132 wait complete. 2023-11-16T23:58:43.093 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 -- 172.21.15.23:0/1708856804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 msgr2=0x7fca2c103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 --2- 172.21.15.23:0/1708856804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c103170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fca28009990 tx=0x7fca28009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 -- 172.21.15.23:0/1708856804 shutdown_connections 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 --2- 172.21.15.23:0/1708856804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c105fd0 0x7fca2c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 --2- 172.21.15.23:0/1708856804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c103170 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 --2- 172.21.15.23:0/1708856804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c1036b0 0x7fca2c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 -- 172.21.15.23:0/1708856804 >> 172.21.15.23:0/1708856804 conn(0x7fca2c0fa750 msgr2=0x7fca2c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 -- 172.21.15.23:0/1708856804 shutdown_connections 2023-11-16T23:58:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.308+0000 7fca335ff700 1 -- 172.21.15.23:0/1708856804 wait complete. 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.309+0000 7fca335ff700 1 Processor -- start 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.309+0000 7fca335ff700 1 -- start start 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 0x7fca2c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c105fd0 0x7fca2c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c19fd80 con 0x7fca2c1036b0 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c19fec0 con 0x7fca2c100d90 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca335ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1a0050 con 0x7fca2c105fd0 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca3139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 0x7fca2c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca31b9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c105fd0 0x7fca2c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca3139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47628/0 (socket says 172.21.15.23:47628) 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca3139b700 1 -- 172.21.15.23:0/3516536512 learned_addr learned my addr 172.21.15.23:0/3516536512 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 -- 172.21.15.23:0/3516536512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c105fd0 msgr2=0x7fca2c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c105fd0 0x7fca2c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 -- 172.21.15.23:0/3516536512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 msgr2=0x7fca2c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.310+0000 7fca30b9a700 1 -- 172.21.15.23:0/3516536512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24009c40 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca30b9a700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 0x7fca2c198740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9f0 tx=0x7fca1c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00c9c0 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca1c010430 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c018680 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca280095f0 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.311+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c10bc40 con 0x7fca2c1036b0 2023-11-16T23:58:43.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.312+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c107b40 con 0x7fca2c1036b0 2023-11-16T23:58:43.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.316+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fca1c010860 con 0x7fca2c1036b0 2023-11-16T23:58:43.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.317+0000 7fca227fc700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 0x7fca180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.317+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fca1c014070 con 0x7fca2c1036b0 2023-11-16T23:58:43.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.317+0000 7fca3139b700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 0x7fca180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.318+0000 7fca3139b700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 0x7fca180a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b2e0 tx=0x7fca28005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.319+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca1c0962d0 con 0x7fca2c1036b0 2023-11-16T23:58:43.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:43 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3539881839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:43 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1884773651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:43 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1609381132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:43.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.537+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fca2c067a60 con 0x7fca2c1036b0 2023-11-16T23:58:43.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.538+0000 7fca227fc700 1 -- 172.21.15.23:0/3516536512 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) v1 ==== 84+0+65 (secure 0 0 0) 0x7fca1c0998f0 con 0x7fca2c1036b0 2023-11-16T23:58:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-16T23:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:] 2023-11-16T23:58:43.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.540+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 msgr2=0x7fca180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.540+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 0x7fca180a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b2e0 tx=0x7fca28005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 msgr2=0x7fca2c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 0x7fca2c198740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00e9f0 tx=0x7fca1c00ed00 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 shutdown_connections 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fca180a11d0 0x7fca180a3680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca2c105fd0 0x7fca2c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca2c100d90 0x7fca2c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 --2- 172.21.15.23:0/3516536512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca2c1036b0 0x7fca2c198740 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 >> 172.21.15.23:0/3516536512 conn(0x7fca2c0fa750 msgr2=0x7fca2c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:43.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 shutdown_connections 2023-11-16T23:58:43.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.541+0000 7fca335ff700 1 -- 172.21.15.23:0/3516536512 wait complete. 2023-11-16T23:58:43.564 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-16T23:58:43.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-16T23:58:43.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:43 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3539881839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:43 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1884773651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:43 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1609381132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:43.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:43 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3539881839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:43 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1884773651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:43.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:43 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1609381132' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:58:43.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.796+0000 7f5cdbdff700 1 -- 172.21.15.23:0/2118121331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4100de0 msgr2=0x7f5cd41031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.796+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/2118121331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4100de0 0x7f5cd41031c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009990 tx=0x7f5cd0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 -- 172.21.15.23:0/2118121331 shutdown_connections 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/2118121331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4106020 0x7f5cd4108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/2118121331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4100de0 0x7f5cd41031c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/2118121331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4103700 0x7f5cd4105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 -- 172.21.15.23:0/2118121331 >> 172.21.15.23:0/2118121331 conn(0x7f5cd40fa7c0 msgr2=0x7f5cd40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 -- 172.21.15.23:0/2118121331 shutdown_connections 2023-11-16T23:58:43.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.797+0000 7f5cdbdff700 1 -- 172.21.15.23:0/2118121331 wait complete. 2023-11-16T23:58:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 Processor -- start 2023-11-16T23:58:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 -- start start 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4103700 0x7f5cd4198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 0x7f5cd419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd419fe40 con 0x7f5cd4106020 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd419ff80 con 0x7f5cd4100de0 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.798+0000 7f5cdbdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd41a0110 con 0x7f5cd4103700 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cda39c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 0x7f5cd419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47648/0 (socket says 172.21.15.23:47648) 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 -- 172.21.15.23:0/3106912949 learned_addr learned my addr 172.21.15.23:0/3106912949 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd939a700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4103700 0x7f5cd4198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 -- 172.21.15.23:0/3106912949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4103700 msgr2=0x7f5cd4198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4103700 0x7f5cd4198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 -- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 msgr2=0x7f5cd419f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 0x7f5cd419f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 -- 172.21.15.23:0/3106912949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd00095f0 con 0x7f5cd4100de0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cda39c700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 0x7f5cd419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.799+0000 7f5cd9b9b700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000b680 tx=0x7f5cd0009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.800+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd001d070 con 0x7f5cd4100de0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.800+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cd0021410 con 0x7f5cd4100de0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.800+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0017650 con 0x7f5cd4100de0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.800+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd410bc70 con 0x7f5cd4100de0 2023-11-16T23:58:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.800+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd410bae0 con 0x7f5cd4100de0 2023-11-16T23:58:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.801+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd4107b90 con 0x7f5cd4100de0 2023-11-16T23:58:43.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.805+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5cd0021580 con 0x7f5cd4100de0 2023-11-16T23:58:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.806+0000 7f5ccaffd700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 0x7f5cc00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.806+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5cd00cd2b0 con 0x7f5cd4100de0 2023-11-16T23:58:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.808+0000 7f5cd939a700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 0x7f5cc00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.809+0000 7f5cd939a700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 0x7f5cc00a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009c60 tx=0x7f5cc4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:43.810+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5cd0096f30 con 0x7f5cd4100de0 2023-11-16T23:58:44.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.028+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5cd4061700 con 0x7f5cd4100de0 2023-11-16T23:58:44.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.697+0000 7f5ccaffd700 1 -- 172.21.15.23:0/3106912949 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5cd009a550 con 0x7f5cd4100de0 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 msgr2=0x7f5cc00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 0x7f5cc00a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009c60 tx=0x7f5cc4009260 comp rx=0 tx=0).stop 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 msgr2=0x7f5cd41982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000b680 tx=0x7f5cd0009580 comp rx=0 tx=0).stop 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 shutdown_connections 2023-11-16T23:58:44.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd4106020 0x7f5cd419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cd4100de0 0x7f5cd41982c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5cc00a1150 0x7f5cc00a3600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 --2- 172.21.15.23:0/3106912949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cd4103700 0x7f5cd4198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.700+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 >> 172.21.15.23:0/3106912949 conn(0x7f5cd40fa7c0 msgr2=0x7f5cd40ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.701+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 shutdown_connections 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.701+0000 7f5cdbdff700 1 -- 172.21.15.23:0/3106912949 wait complete. 2023-11-16T23:58:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2023-11-16T23:58:44.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-16T23:58:44.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:44 smithi173 ceph-mon[131534]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:44.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:44 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3516536512' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T23:58:44.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:44 smithi156 ceph-mon[131105]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:44.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:44 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3516536512' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T23:58:44.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 -- 172.21.15.23:0/2636229871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81002d0 msgr2=0x7fe5c81006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:44.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/2636229871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81002d0 0x7fe5c81006e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40098f0 tx=0x7fe5c4009c00 comp rx=0 tx=0).stop 2023-11-16T23:58:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 -- 172.21.15.23:0/2636229871 shutdown_connections 2023-11-16T23:58:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/2636229871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 0x7fe5c8108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/2636229871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81002d0 0x7fe5c81006e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/2636229871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81014c0 0x7fe5c8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 -- 172.21.15.23:0/2636229871 >> 172.21.15.23:0/2636229871 conn(0x7fe5c80fb8b0 msgr2=0x7fe5c80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:44.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 -- 172.21.15.23:0/2636229871 shutdown_connections 2023-11-16T23:58:44.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.939+0000 7fe5cfdff700 1 -- 172.21.15.23:0/2636229871 wait complete. 2023-11-16T23:58:44.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 Processor -- start 2023-11-16T23:58:44.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 -- start start 2023-11-16T23:58:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 0x7fe5c8198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 0x7fe5c819f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c819fee0 con 0x7fe5c81014c0 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c81a0020 con 0x7fe5c8101e50 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.940+0000 7fe5cfdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c81a01b0 con 0x7fe5c81002d0 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cdb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 0x7fe5c8198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40768/0 (socket says 172.21.15.23:40768) 2023-11-16T23:58:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 -- 172.21.15.23:0/1552729580 learned_addr learned my addr 172.21.15.23:0/1552729580 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5ce39c700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 0x7fe5c819f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 -- 172.21.15.23:0/1552729580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 msgr2=0x7fe5c8198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 0x7fe5c8198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 -- 172.21.15.23:0/1552729580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 msgr2=0x7fe5c819f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 0x7fe5c819f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 -- 172.21.15.23:0/1552729580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c4009520 con 0x7fe5c81014c0 2023-11-16T23:58:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cdb9b700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 0x7fe5c8198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.941+0000 7fe5cd39a700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800e9c0 tx=0x7fe5b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.942+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8004a30 con 0x7fe5c81014c0 2023-11-16T23:58:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.942+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5b800cc80 con 0x7fe5c81014c0 2023-11-16T23:58:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.942+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80054e0 con 0x7fe5c81014c0 2023-11-16T23:58:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.942+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c810bca0 con 0x7fe5c81014c0 2023-11-16T23:58:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.942+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c810bb10 con 0x7fe5c81014c0 2023-11-16T23:58:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.943+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c8192a00 con 0x7fe5c81014c0 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.945+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe5b8004b90 con 0x7fe5c81014c0 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.946+0000 7fe5beffd700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 0x7fe5b40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.947+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fe5b8014070 con 0x7fe5c81014c0 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.947+0000 7fe5cdb9b700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 0x7fe5b40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.947+0000 7fe5cdb9b700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 0x7fe5b40a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009fd0 tx=0x7fe5c4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:44.952+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe5b8096310 con 0x7fe5c81014c0 2023-11-16T23:58:45.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:44 smithi023 ceph-mon[97824]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:45.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:44 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3516536512' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T23:58:45.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.166+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fe5c802cfe0 con 0x7fe5c81014c0 2023-11-16T23:58:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.696+0000 7fe5beffd700 1 -- 172.21.15.23:0/1552729580 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v82) v1 ==== 171+0+0 (secure 0 0 0) 0x7fe5b8099930 con 0x7fe5c81014c0 2023-11-16T23:58:45.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.702+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 msgr2=0x7fe5b40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:45.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 0x7fe5b40a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009fd0 tx=0x7fe5c4019040 comp rx=0 tx=0).stop 2023-11-16T23:58:45.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 msgr2=0x7fe5c81988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:45.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800e9c0 tx=0x7fe5b800ecd0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 shutdown_connections 2023-11-16T23:58:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5c8101e50 0x7fe5c819f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe5b40a1150 0x7fe5b40a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5c81002d0 0x7fe5c8198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 --2- 172.21.15.23:0/1552729580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c81014c0 0x7fe5c81988a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.703+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 >> 172.21.15.23:0/1552729580 conn(0x7fe5c80fb8b0 msgr2=0x7fe5c8106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:45.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.704+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 shutdown_connections 2023-11-16T23:58:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.704+0000 7fe5cfdff700 1 -- 172.21.15.23:0/1552729580 wait complete. 2023-11-16T23:58:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2023-11-16T23:58:45.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 -- 172.21.15.23:0/4063996281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 msgr2=0x7fe2981031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 --2- 172.21.15.23:0/4063996281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe2981031e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe288009990 tx=0x7fe288009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 -- 172.21.15.23:0/4063996281 shutdown_connections 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 --2- 172.21.15.23:0/4063996281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 0x7fe298108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 --2- 172.21.15.23:0/4063996281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe2981031e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 --2- 172.21.15.23:0/4063996281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 0x7fe298105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 -- 172.21.15.23:0/4063996281 >> 172.21.15.23:0/4063996281 conn(0x7fe2980fa7c0 msgr2=0x7fe2980fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:45.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 -- 172.21.15.23:0/4063996281 shutdown_connections 2023-11-16T23:58:45.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.979+0000 7fe29dfff700 1 -- 172.21.15.23:0/4063996281 wait complete. 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.980+0000 7fe29dfff700 1 Processor -- start 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 -- start start 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 0x7fe298198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 0x7fe29819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29819fea0 con 0x7fe298100e00 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29819ffe0 con 0x7fe298103720 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe29dfff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2981a0170 con 0x7fe298106040 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe2977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe2977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40794/0 (socket says 172.21.15.23:40794) 2023-11-16T23:58:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe2977fe700 1 -- 172.21.15.23:0/2228241545 learned_addr learned my addr 172.21.15.23:0/2228241545 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe297fff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 0x7fe29819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.981+0000 7fe296ffd700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 0x7fe298198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 -- 172.21.15.23:0/2228241545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 msgr2=0x7fe29819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 0x7fe29819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 -- 172.21.15.23:0/2228241545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 msgr2=0x7fe298198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 0x7fe298198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 -- 172.21.15.23:0/2228241545 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2880095f0 con 0x7fe298100e00 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe2977fe700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b680 tx=0x7fe288009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28801d070 con 0x7fe298100e00 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29810bcc0 con 0x7fe298100e00 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe288021410 con 0x7fe298100e00 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe288017650 con 0x7fe298100e00 2023-11-16T23:58:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.982+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29810bb30 con 0x7fe298100e00 2023-11-16T23:58:45.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.983+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe298107bb0 con 0x7fe298100e00 2023-11-16T23:58:45.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.986+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe288021580 con 0x7fe298100e00 2023-11-16T23:58:45.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.987+0000 7fe294ff9700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 0x7fe2800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:45.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.987+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fe2880cdbd0 con 0x7fe298100e00 2023-11-16T23:58:45.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.987+0000 7fe296ffd700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 0x7fe2800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:45.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.988+0000 7fe296ffd700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 0x7fe2800a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009c60 tx=0x7fe28c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:45.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:45.990+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe2880974c0 con 0x7fe298100e00 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3106912949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T23:58:46.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:45 smithi023 ceph-mon[97824]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T23:58:46.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3106912949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T23:58:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:45 smithi173 ceph-mon[131534]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T23:58:46.111 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3106912949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.111 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T23:58:46.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.113 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.113 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:46.113 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T23:58:46.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:45 smithi156 ceph-mon[131105]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T23:58:46.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.250+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe29802cfe0 con 0x7fe298100e00 2023-11-16T23:58:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.265+0000 7fe294ff9700 1 -- 172.21.15.23:0/2228241545 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe28809aae0 con 0x7fe298100e00 2023-11-16T23:58:46.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.270+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 msgr2=0x7fe2800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:46.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.270+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 0x7fe2800a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009c60 tx=0x7fe28c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.270+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 msgr2=0x7fe298198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.270+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b680 tx=0x7fe288009580 comp rx=0 tx=0).stop 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 shutdown_connections 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2800a1150 0x7fe2800a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe298106040 0x7fe29819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe298100e00 0x7fe298198320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 --2- 172.21.15.23:0/2228241545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe298103720 0x7fe298198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 >> 172.21.15.23:0/2228241545 conn(0x7fe2980fa7c0 msgr2=0x7fe2980ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:46.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.271+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 shutdown_connections 2023-11-16T23:58:46.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.272+0000 7fe29dfff700 1 -- 172.21.15.23:0/2228241545 wait complete. 2023-11-16T23:58:46.282 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2023-11-16T23:58:46.309 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-16T23:58:46.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 -- 172.21.15.23:0/3539231469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 msgr2=0x7fb1981006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/3539231469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981006e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb188009990 tx=0x7fb188009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 -- 172.21.15.23:0/3539231469 shutdown_connections 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/3539231469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 0x7fb198108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/3539231469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981006e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/3539231469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 0x7fb198101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 -- 172.21.15.23:0/3539231469 >> 172.21.15.23:0/3539231469 conn(0x7fb1980fb8b0 msgr2=0x7fb1980fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 -- 172.21.15.23:0/3539231469 shutdown_connections 2023-11-16T23:58:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.524+0000 7fb19e5ff700 1 -- 172.21.15.23:0/3539231469 wait complete. 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 Processor -- start 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 -- start start 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 0x7fb198198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 0x7fb19819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19819fe40 con 0x7fb1981002d0 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19819ff80 con 0x7fb1981014c0 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.525+0000 7fb19e5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1981a0110 con 0x7fb198101e50 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40798/0 (socket says 172.21.15.23:40798) 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 -- 172.21.15.23:0/4097692109 learned_addr learned my addr 172.21.15.23:0/4097692109 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:46.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb1977fe700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 0x7fb198198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb19cb9c700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 0x7fb19819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 -- 172.21.15.23:0/4097692109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 msgr2=0x7fb19819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 0x7fb19819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 -- 172.21.15.23:0/4097692109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 msgr2=0x7fb198198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 0x7fb198198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 -- 172.21.15.23:0/4097692109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880095f0 con 0x7fb1981002d0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.526+0000 7fb197fff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb188005740 tx=0x7fb188005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.527+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18801d070 con 0x7fb1981002d0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.527+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19810bca0 con 0x7fb1981002d0 2023-11-16T23:58:46.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.527+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb188021410 con 0x7fb1981002d0 2023-11-16T23:58:46.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.527+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188017650 con 0x7fb1981002d0 2023-11-16T23:58:46.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.527+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19810bb10 con 0x7fb1981002d0 2023-11-16T23:58:46.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.528+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb198067b00 con 0x7fb1981002d0 2023-11-16T23:58:46.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.531+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb188021580 con 0x7fb1981002d0 2023-11-16T23:58:46.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.532+0000 7fb1957fa700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 0x7fb1800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:46.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.532+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb1880cde00 con 0x7fb1981002d0 2023-11-16T23:58:46.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.532+0000 7fb1977fe700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 0x7fb1800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:46.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.533+0000 7fb1977fe700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 0x7fb1800a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009c60 tx=0x7fb18c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:46.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.534+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb188097690 con 0x7fb1981002d0 2023-11-16T23:58:46.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:46.758+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fb1981a0b90 con 0x7fb1981002d0 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T23:58:47.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T23:58:47.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T23:58:47.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T23:58:47.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:46 smithi023 ceph-mon[97824]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T23:58:47.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T23:58:47.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:46 smithi173 ceph-mon[131534]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: pgmap v288: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1552729580' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T23:58:47.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2228241545' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T23:58:47.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:46 smithi156 ceph-mon[131105]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T23:58:47.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.267+0000 7fb1957fa700 1 -- 172.21.15.23:0/4097692109 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v84) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb18809acb0 con 0x7fb1981002d0 2023-11-16T23:58:47.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 msgr2=0x7fb1800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:47.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 0x7fb1800a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009c60 tx=0x7fb18c009260 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 msgr2=0x7fb1981982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb188005740 tx=0x7fb188005770 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 shutdown_connections 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb1800a1150 0x7fb1800a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb198101e50 0x7fb19819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1981002d0 0x7fb1981982c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 --2- 172.21.15.23:0/4097692109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1981014c0 0x7fb198198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 >> 172.21.15.23:0/4097692109 conn(0x7fb1980fb8b0 msgr2=0x7fb1980fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 shutdown_connections 2023-11-16T23:58:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.270+0000 7fb19e5ff700 1 -- 172.21.15.23:0/4097692109 wait complete. 2023-11-16T23:58:47.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-16T23:58:47.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/738173289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 msgr2=0x7fbc18101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/738173289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18101920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009990 tx=0x7fbc0c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/738173289 shutdown_connections 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/738173289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 0x7fbc18108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/738173289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc181006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/738173289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18101920 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/738173289 >> 172.21.15.23:0/738173289 conn(0x7fbc180fb8c0 msgr2=0x7fbc180fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/738173289 shutdown_connections 2023-11-16T23:58:47.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.511+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/738173289 wait complete. 2023-11-16T23:58:47.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 Processor -- start 2023-11-16T23:58:47.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 -- start start 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 0x7fbc18073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18073bc0 con 0x7fbc181002e0 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18073d30 con 0x7fbc181014d0 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.512+0000 7fbc1e5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc181a0080 con 0x7fbc18101e60 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40806/0 (socket says 172.21.15.23:40806) 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 -- 172.21.15.23:0/3347827870 learned_addr learned my addr 172.21.15.23:0/3347827870 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc177fe700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc1cb9c700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 0x7fbc18073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:47.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 -- 172.21.15.23:0/3347827870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 msgr2=0x7fbc18073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 0x7fbc18073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 -- 172.21.15.23:0/3347827870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 msgr2=0x7fbc18072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 -- 172.21.15.23:0/3347827870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0095f0 con 0x7fbc181002e0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc17fff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e970 tx=0x7fbc0800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08009790 con 0x7fbc181002e0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1810bc50 con 0x7fbc181002e0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc0800c7e0 con 0x7fbc181002e0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.513+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08003c70 con 0x7fbc181002e0 2023-11-16T23:58:47.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.514+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1810bac0 con 0x7fbc181002e0 2023-11-16T23:58:47.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.514+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc181a07d0 con 0x7fbc181002e0 2023-11-16T23:58:47.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.517+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbc0800cc50 con 0x7fbc181002e0 2023-11-16T23:58:47.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.518+0000 7fbc157fa700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 0x7fbc000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:47.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.518+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fbc08014070 con 0x7fbc181002e0 2023-11-16T23:58:47.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.518+0000 7fbc177fe700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 0x7fbc000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:47.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.519+0000 7fbc177fe700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 0x7fbc000a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00b2e0 tx=0x7fbc0c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:47.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.520+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbc08096370 con 0x7fbc181002e0 2023-11-16T23:58:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:47.736+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbc18107190 con 0x7fbc181002e0 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: pgmap v291: 97 pgs: 3 creating+peering, 25 active+clean, 69 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2023-11-16T23:58:48.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:active 2023-11-16T23:58:48.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:48.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T23:58:48.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T23:58:48.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:47 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:48.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: pgmap v291: 97 pgs: 3 creating+peering, 25 active+clean, 69 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2023-11-16T23:58:48.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:active 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:47 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: pgmap v291: 97 pgs: 3 creating+peering, 25 active+clean, 69 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] up:active 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4097692109' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T23:58:48.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:47 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:49.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:48 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T23:58:49.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:48 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T23:58:49.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:48 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T23:58:49.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.710+0000 7fbc157fa700 1 -- 172.21.15.23:0/3347827870 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v86) v1 ==== 228+0+0 (secure 0 0 0) 0x7fbc08099990 con 0x7fbc181002e0 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.718+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 msgr2=0x7fbc000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.718+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 0x7fbc000a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00b2e0 tx=0x7fbc0c005960 comp rx=0 tx=0).stop 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.718+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 msgr2=0x7fbc180748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.718+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e970 tx=0x7fbc0800ec80 comp rx=0 tx=0).stop 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 shutdown_connections 2023-11-16T23:58:49.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbc000a1150 0x7fbc000a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc18101e60 0x7fbc18073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc181002e0 0x7fbc180748a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 --2- 172.21.15.23:0/3347827870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc181014d0 0x7fbc18072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 >> 172.21.15.23:0/3347827870 conn(0x7fbc180fb8c0 msgr2=0x7fbc180fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 shutdown_connections 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.719+0000 7fbc1e5ff700 1 -- 172.21.15.23:0/3347827870 wait complete. 2023-11-16T23:58:49.722 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2023-11-16T23:58:49.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-16T23:58:49.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 -- 172.21.15.23:0/2558745451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3294100050 msgr2=0x7f3294100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:49.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 --2- 172.21.15.23:0/2558745451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3294100050 0x7f3294100460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3284009990 tx=0x7f3284009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 -- 172.21.15.23:0/2558745451 shutdown_connections 2023-11-16T23:58:49.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 --2- 172.21.15.23:0/2558745451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294101c90 0x7f3294108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 --2- 172.21.15.23:0/2558745451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3294100050 0x7f3294100460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 --2- 172.21.15.23:0/2558745451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32941012e0 0x7f3294101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 -- 172.21.15.23:0/2558745451 >> 172.21.15.23:0/2558745451 conn(0x7f32940fb650 msgr2=0x7f32940fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:49.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 -- 172.21.15.23:0/2558745451 shutdown_connections 2023-11-16T23:58:49.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.956+0000 7f329919d700 1 -- 172.21.15.23:0/2558745451 wait complete. 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.957+0000 7f329919d700 1 Processor -- start 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 -- start start 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 0x7f32941962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3294101c90 0x7f329419d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329419d8e0 con 0x7f3294101c90 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329419da50 con 0x7f32941012e0 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329919d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329419dc10 con 0x7f3294100050 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f3293fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:49.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f32937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 0x7f32941962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f3293fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:49438/0 (socket says 172.21.15.23:49438) 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f3293fff700 1 -- 172.21.15.23:0/728976413 learned_addr learned my addr 172.21.15.23:0/728976413 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.958+0000 7f329899c700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3294101c90 0x7f329419d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 -- 172.21.15.23:0/728976413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 msgr2=0x7f3294195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 -- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3294101c90 msgr2=0x7f329419d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3294101c90 0x7f329419d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 -- 172.21.15.23:0/728976413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840095f0 con 0x7f32941012e0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f3293fff700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32937fe700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 0x7f32941962e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f328800e9f0 tx=0x7f328800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328800c9c0 con 0x7f32941012e0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329410ba00 con 0x7f32941012e0 2023-11-16T23:58:49.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3288010430 con 0x7f32941012e0 2023-11-16T23:58:49.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.959+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288018680 con 0x7f32941012e0 2023-11-16T23:58:49.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.960+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329410b870 con 0x7f32941012e0 2023-11-16T23:58:49.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.961+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329419e250 con 0x7f32941012e0 2023-11-16T23:58:49.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.962+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3288010860 con 0x7f32941012e0 2023-11-16T23:58:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.964+0000 7f32917fa700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 0x7f327c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.964+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f3288014070 con 0x7f32941012e0 2023-11-16T23:58:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.964+0000 7f3293fff700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 0x7f327c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.964+0000 7f3293fff700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 0x7f327c0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3294197270 tx=0x7f328400b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:49.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:49.968+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3288051730 con 0x7f32941012e0 2023-11-16T23:58:50.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:49 smithi023 ceph-mon[97824]: pgmap v294: 97 pgs: 3 creating+peering, 71 active+clean, 23 unknown; 577 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T23:58:50.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:49 smithi023 ceph-mon[97824]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T23:58:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:49 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:49 smithi173 ceph-mon[131534]: pgmap v294: 97 pgs: 3 creating+peering, 71 active+clean, 23 unknown; 577 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T23:58:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:49 smithi173 ceph-mon[131534]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T23:58:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:49 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:49 smithi156 ceph-mon[131105]: pgmap v294: 97 pgs: 3 creating+peering, 71 active+clean, 23 unknown; 577 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-11-16T23:58:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:49 smithi156 ceph-mon[131105]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T23:58:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:49 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.214+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f3294106fa0 con 0x7f32941012e0 2023-11-16T23:58:50.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.712+0000 7f32917fa700 1 -- 172.21.15.23:0/728976413 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3288096180 con 0x7f32941012e0 2023-11-16T23:58:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 msgr2=0x7f327c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 0x7f327c0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3294197270 tx=0x7f328400b260 comp rx=0 tx=0).stop 2023-11-16T23:58:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 msgr2=0x7f32941962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 0x7f32941962e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f328800e9f0 tx=0x7f328800ed00 comp rx=0 tx=0).stop 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 shutdown_connections 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3294101c90 0x7f329419d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f327c0a1150 0x7f327c0a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3294100050 0x7f3294195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 --2- 172.21.15.23:0/728976413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32941012e0 0x7f32941962e0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f328800e9f0 tx=0x7f328800ed00 comp rx=0 tx=0).stop 2023-11-16T23:58:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.719+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 >> 172.21.15.23:0/728976413 conn(0x7f32940fb650 msgr2=0x7f32940fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:50.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.720+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 shutdown_connections 2023-11-16T23:58:50.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.720+0000 7f329919d700 1 -- 172.21.15.23:0/728976413 wait complete. 2023-11-16T23:58:50.730 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2023-11-16T23:58:50.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 -- 172.21.15.23:0/1168318029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841014b0 msgr2=0x7fc584101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/1168318029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841014b0 0x7fc584101900 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc574009990 tx=0x7fc574009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 -- 172.21.15.23:0/1168318029 shutdown_connections 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/1168318029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/1168318029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841002c0 0x7fc5841006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/1168318029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841014b0 0x7fc584101900 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 -- 172.21.15.23:0/1168318029 >> 172.21.15.23:0/1168318029 conn(0x7fc5840fb8a0 msgr2=0x7fc5840fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 -- 172.21.15.23:0/1168318029 shutdown_connections 2023-11-16T23:58:50.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.965+0000 7fc58b3ff700 1 -- 172.21.15.23:0/1168318029 wait complete. 2023-11-16T23:58:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.966+0000 7fc58b3ff700 1 Processor -- start 2023-11-16T23:58:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 -- start start 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841002c0 0x7fc5840748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841014b0 0x7fc584072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584073bc0 con 0x7fc584101e40 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584073d30 con 0x7fc5841014b0 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58b3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5841a0080 con 0x7fc5841002c0 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.967+0000 7fc58999c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56394/0 (socket says 172.21.15.23:56394) 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 -- 172.21.15.23:0/992127166 learned_addr learned my addr 172.21.15.23:0/992127166 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58919b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841002c0 0x7fc5840748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58899a700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841014b0 0x7fc584072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 -- 172.21.15.23:0/992127166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841002c0 msgr2=0x7fc5840748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841002c0 0x7fc5840748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 -- 172.21.15.23:0/992127166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841014b0 msgr2=0x7fc584072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841014b0 0x7fc584072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 -- 172.21.15.23:0/992127166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740095f0 con 0x7fc584101e40 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58999c700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00ef30 tx=0x7fc57c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c018070 con 0x7fc584101e40 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc57c00f040 con 0x7fc584101e40 2023-11-16T23:58:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c00cd00 con 0x7fc584101e40 2023-11-16T23:58:50.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.968+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58410bc20 con 0x7fc584101e40 2023-11-16T23:58:50.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.969+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58410ba90 con 0x7fc584101e40 2023-11-16T23:58:50.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.970+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc584107860 con 0x7fc584101e40 2023-11-16T23:58:50.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.972+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc57c003a00 con 0x7fc584101e40 2023-11-16T23:58:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.973+0000 7fc57a7fc700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 0x7fc5700a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.973+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fc57c0ccf40 con 0x7fc584101e40 2023-11-16T23:58:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.973+0000 7fc58919b700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 0x7fc5700a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.974+0000 7fc58919b700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 0x7fc5700a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc584101310 tx=0x7fc580009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:50.976+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc57c0964f0 con 0x7fc584101e40 2023-11-16T23:58:51.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/728976413' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:51.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T23:58:51.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/728976413' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:51.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3347827870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/728976413' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T23:58:51.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T23:58:51.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:58:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.231+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fc58402cfe0 con 0x7fc584101e40 2023-11-16T23:58:51.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.242+0000 7fc57a7fc700 1 -- 172.21.15.23:0/992127166 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fc57c099b10 con 0x7fc584101e40 2023-11-16T23:58:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 msgr2=0x7fc5700a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 0x7fc5700a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc584101310 tx=0x7fc580009260 comp rx=0 tx=0).stop 2023-11-16T23:58:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 msgr2=0x7fc584073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00ef30 tx=0x7fc57c00c3a0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 shutdown_connections 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc584101e40 0x7fc584073420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5700a11d0 0x7fc5700a3680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5841002c0 0x7fc5840748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 --2- 172.21.15.23:0/992127166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5841014b0 0x7fc584072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.244+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 >> 172.21.15.23:0/992127166 conn(0x7fc5840fb8a0 msgr2=0x7fc5841049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.245+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 shutdown_connections 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.245+0000 7fc58b3ff700 1 -- 172.21.15.23:0/992127166 wait complete. 2023-11-16T23:58:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 4 to fsmap 2023-11-16T23:58:51.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 -- 172.21.15.23:0/613595787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 msgr2=0x7f9f401006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 --2- 172.21.15.23:0/613595787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f401006a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009990 tx=0x7f9f30009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 -- 172.21.15.23:0/613595787 shutdown_connections 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 --2- 172.21.15.23:0/613595787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f401086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 --2- 172.21.15.23:0/613595787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f401006a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 --2- 172.21.15.23:0/613595787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 0x7f9f401018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 -- 172.21.15.23:0/613595787 >> 172.21.15.23:0/613595787 conn(0x7f9f400fb870 msgr2=0x7f9f400fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 -- 172.21.15.23:0/613595787 shutdown_connections 2023-11-16T23:58:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.467+0000 7f9f461ff700 1 -- 172.21.15.23:0/613595787 wait complete. 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.468+0000 7f9f461ff700 1 Processor -- start 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.468+0000 7f9f461ff700 1 -- start start 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f40196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 0x7f9f40196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4019db80 con 0x7f9f40101e10 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4019dcc0 con 0x7f9f40100290 2023-11-16T23:58:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f461ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4019de50 con 0x7f9f40101480 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56418/0 (socket says 172.21.15.23:56418) 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 -- 172.21.15.23:0/2146560557 learned_addr learned my addr 172.21.15.23:0/2146560557 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3f7fe700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f40196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3effd700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 0x7f9f40196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 -- 172.21.15.23:0/2146560557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 msgr2=0x7f9f40196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 0x7f9f40196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 -- 172.21.15.23:0/2146560557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 msgr2=0x7f9f40196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.469+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f40196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3ffff700 1 -- 172.21.15.23:0/2146560557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300095f0 con 0x7f9f40101e10 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3f7fe700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f40196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800ef30 tx=0x7f9f3800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38018070 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4010bc40 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f3800f040 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800cd00 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.470+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4010bab0 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.471+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4010bdd0 con 0x7f9f40101e10 2023-11-16T23:58:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.473+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f38003a00 con 0x7f9f40101e10 2023-11-16T23:58:51.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.475+0000 7f9f3cff9700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 0x7f9f280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.475+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9f380ccfd0 con 0x7f9f40101e10 2023-11-16T23:58:51.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.475+0000 7f9f3f7fe700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 0x7f9f280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.475+0000 7f9f3f7fe700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 0x7f9f280a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000b2e0 tx=0x7f9f30005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:51.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.477+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f38096560 con 0x7f9f40101e10 2023-11-16T23:58:51.607 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: 2023-11-16T23:58:51.137+0000 7fac3172e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-16T23:58:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.728+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f4019ea10 con 0x7f9f40101e10 2023-11-16T23:58:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.728+0000 7f9f3cff9700 1 -- 172.21.15.23:0/2146560557 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10753 (secure 0 0 0) 0x7f9f38099b80 con 0x7f9f40101e10 2023-11-16T23:58:51.731 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:51.732 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:58:51.234117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T23:58:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 msgr2=0x7f9f280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 0x7f9f280a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000b2e0 tx=0x7f9f30005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 msgr2=0x7f9f4019d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800ef30 tx=0x7f9f3800c3a0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 shutdown_connections 2023-11-16T23:58:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40101e10 0x7f9f4019d5b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f40100290 0x7f9f40196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f280a1150 0x7f9f280a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.731+0000 7f9f461ff700 1 --2- 172.21.15.23:0/2146560557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f40101480 0x7f9f40196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.732+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 >> 172.21.15.23:0/2146560557 conn(0x7f9f400fb870 msgr2=0x7f9f400fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.732+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 shutdown_connections 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.732+0000 7f9f461ff700 1 -- 172.21.15.23:0/2146560557 wait complete. 2023-11-16T23:58:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2023-11-16T23:58:51.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:58:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 -- 172.21.15.23:0/703078192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1002b0 msgr2=0x7fbe6c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 --2- 172.21.15.23:0/703078192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c1006c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009990 tx=0x7fbe5c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 -- 172.21.15.23:0/703078192 shutdown_connections 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 --2- 172.21.15.23:0/703078192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 0x7fbe6c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 --2- 172.21.15.23:0/703078192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c1006c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 --2- 172.21.15.23:0/703078192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 -- 172.21.15.23:0/703078192 >> 172.21.15.23:0/703078192 conn(0x7fbe6c0fb890 msgr2=0x7fbe6c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 -- 172.21.15.23:0/703078192 shutdown_connections 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.958+0000 7fbe715ff700 1 -- 172.21.15.23:0/703078192 wait complete. 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 Processor -- start 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 -- start start 2023-11-16T23:58:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 0x7fbe6c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c19dbc0 con 0x7fbe6c1014a0 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c19dd00 con 0x7fbe6c101e30 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.959+0000 7fbe715ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c19de90 con 0x7fbe6c1002b0 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56438/0 (socket says 172.21.15.23:56438) 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 -- 172.21.15.23:0/35293516 learned_addr learned my addr 172.21.15.23:0/35293516 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6affd700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6b7fe700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 0x7fbe6c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 -- 172.21.15.23:0/35293516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1002b0 msgr2=0x7fbe6c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 -- 172.21.15.23:0/35293516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 msgr2=0x7fbe6c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 0x7fbe6c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 -- 172.21.15.23:0/35293516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0095f0 con 0x7fbe6c1014a0 2023-11-16T23:58:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe6a7fc700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000d950 tx=0x7fbe6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.960+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6000f890 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.961+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe60010460 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.961+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6c10bc80 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.961+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6000e610 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.961+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6c10baf0 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.962+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6c1078c0 con 0x7fbe6c1014a0 2023-11-16T23:58:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.964+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbe600108d0 con 0x7fbe6c1014a0 2023-11-16T23:58:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.965+0000 7fbe53fff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 0x7fbe540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.965+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbe600cc940 con 0x7fbe6c1014a0 2023-11-16T23:58:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.965+0000 7fbe6affd700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 0x7fbe540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.966+0000 7fbe6affd700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 0x7fbe540a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00b5c0 tx=0x7fbe5c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:51.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:51.969+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbe60095e40 con 0x7fbe6c1014a0 2023-11-16T23:58:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-16T23:58:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: pgmap v297: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T23:58:52.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:51 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-16T23:58:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: pgmap v297: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:51 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-16T23:58:52.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: pgmap v297: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/992127166' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T23:58:52.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:51 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:58:52.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.184+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe6c02cfe0 con 0x7fbe6c1014a0 2023-11-16T23:58:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.185+0000 7fbe53fff700 1 -- 172.21.15.23:0/35293516 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fbe60016090 con 0x7fbe6c1014a0 2023-11-16T23:58:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:52.188 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 msgr2=0x7fbe540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 0x7fbe540a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00b5c0 tx=0x7fbe5c005fd0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 msgr2=0x7fbe6c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000d950 tx=0x7fbe6000dc60 comp rx=0 tx=0).stop 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 shutdown_connections 2023-11-16T23:58:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe6c101e30 0x7fbe6c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbe540a1150 0x7fbe540a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe6c1002b0 0x7fbe6c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 --2- 172.21.15.23:0/35293516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe6c1014a0 0x7fbe6c196580 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 >> 172.21.15.23:0/35293516 conn(0x7fbe6c0fb890 msgr2=0x7fbe6c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 shutdown_connections 2023-11-16T23:58:52.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.188+0000 7fbe715ff700 1 -- 172.21.15.23:0/35293516 wait complete. 2023-11-16T23:58:52.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:58:52.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.411+0000 7ff12d3ff700 1 -- 172.21.15.23:0/3362976928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128100290 msgr2=0x7ff1281006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.411+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/3362976928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128100290 0x7ff1281006a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff118009990 tx=0x7ff118009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 -- 172.21.15.23:0/3362976928 shutdown_connections 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/3362976928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128101e10 0x7ff1281086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/3362976928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128100290 0x7ff1281006a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/3362976928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101480 0x7ff1281018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 -- 172.21.15.23:0/3362976928 >> 172.21.15.23:0/3362976928 conn(0x7ff1280fb870 msgr2=0x7ff1280fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 -- 172.21.15.23:0/3362976928 shutdown_connections 2023-11-16T23:58:52.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.412+0000 7ff12d3ff700 1 -- 172.21.15.23:0/3362976928 wait complete. 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 Processor -- start 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 -- start start 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128100290 0x7ff1281981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101e10 0x7ff12819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12819fd40 con 0x7ff128101480 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12819fe80 con 0x7ff128101e10 2023-11-16T23:58:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff12d3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1281a0010 con 0x7ff128100290 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff11ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff11ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56448/0 (socket says 172.21.15.23:56448) 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff11ffff700 1 -- 172.21.15.23:0/1941850311 learned_addr learned my addr 172.21.15.23:0/1941850311 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.413+0000 7ff126ffd700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128100290 0x7ff1281981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff1277fe700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101e10 0x7ff12819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 -- 172.21.15.23:0/1941850311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128100290 msgr2=0x7ff1281981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128100290 0x7ff1281981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 -- 172.21.15.23:0/1941850311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101e10 msgr2=0x7ff12819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101e10 0x7ff12819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 -- 172.21.15.23:0/1941850311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180095f0 con 0x7ff128101480 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff11ffff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d980 tx=0x7ff11000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110004220 con 0x7ff128101480 2023-11-16T23:58:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff110007640 con 0x7ff128101480 2023-11-16T23:58:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12810bc40 con 0x7ff128101480 2023-11-16T23:58:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110005690 con 0x7ff128101480 2023-11-16T23:58:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.414+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12810bab0 con 0x7ff128101480 2023-11-16T23:58:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.415+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1281928d0 con 0x7ff128101480 2023-11-16T23:58:52.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.417+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff110004750 con 0x7ff128101480 2023-11-16T23:58:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.419+0000 7ff124ff9700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 0x7ff1140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.419+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff1100cc980 con 0x7ff128101480 2023-11-16T23:58:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.419+0000 7ff126ffd700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 0x7ff1140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.420+0000 7ff126ffd700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 0x7ff1140a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b2e0 tx=0x7ff118005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:52.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.423+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff110095e80 con 0x7ff128101480 2023-11-16T23:58:52.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.633+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12802cfe0 con 0x7ff128101480 2023-11-16T23:58:52.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.635+0000 7ff124ff9700 1 -- 172.21.15.23:0/1941850311 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7ff110016020 con 0x7ff128101480 2023-11-16T23:58:52.636 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:52.637 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:52.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 msgr2=0x7ff1140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 0x7ff1140a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b2e0 tx=0x7ff118005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 msgr2=0x7ff128198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d980 tx=0x7ff11000dc90 comp rx=0 tx=0).stop 2023-11-16T23:58:52.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 shutdown_connections 2023-11-16T23:58:52.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff128101e10 0x7ff12819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1140a1150 0x7ff1140a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff128100290 0x7ff1281981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 --2- 172.21.15.23:0/1941850311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff128101480 0x7ff128198700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 >> 172.21.15.23:0/1941850311 conn(0x7ff1280fb870 msgr2=0x7ff128106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:52.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 shutdown_connections 2023-11-16T23:58:52.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.637+0000 7ff12d3ff700 1 -- 172.21.15.23:0/1941850311 wait complete. 2023-11-16T23:58:52.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:58:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 -- 172.21.15.23:0/1939199403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 msgr2=0x7f61d81018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 --2- 172.21.15.23:0/1939199403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d81018d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009990 tx=0x7f61c8009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 -- 172.21.15.23:0/1939199403 shutdown_connections 2023-11-16T23:58:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 --2- 172.21.15.23:0/1939199403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 0x7f61d81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 --2- 172.21.15.23:0/1939199403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 --2- 172.21.15.23:0/1939199403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d81018d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.875+0000 7f61dffff700 1 -- 172.21.15.23:0/1939199403 >> 172.21.15.23:0/1939199403 conn(0x7f61d80fb870 msgr2=0x7f61d80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.876+0000 7f61dffff700 1 -- 172.21.15.23:0/1939199403 shutdown_connections 2023-11-16T23:58:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.876+0000 7f61dffff700 1 -- 172.21.15.23:0/1939199403 wait complete. 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 Processor -- start 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 -- start start 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 0x7f61d819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d819fd40 con 0x7f61d8101480 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d819fe80 con 0x7f61d8100290 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dffff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d81a0010 con 0x7f61d8101e10 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61dd59a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61ddd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61ddd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57728/0 (socket says 172.21.15.23:57728) 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61ddd9b700 1 -- 172.21.15.23:0/2561991894 learned_addr learned my addr 172.21.15.23:0/2561991894 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.877+0000 7f61de59c700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 0x7f61d819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 -- 172.21.15.23:0/2561991894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 msgr2=0x7f61d819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 0x7f61d819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 -- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 msgr2=0x7f61d8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 -- 172.21.15.23:0/2561991894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80095f0 con 0x7f61d8100290 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ddd9b700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e970 tx=0x7f61d400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4009790 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61d400c7e0 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d810bc50 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4003c70 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.878+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d810bac0 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.879+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d8067a60 con 0x7f61d8100290 2023-11-16T23:58:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.881+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f61d400cc50 con 0x7f61d8100290 2023-11-16T23:58:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.882+0000 7f61ceffd700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 0x7f61c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.883+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f61d4014070 con 0x7f61d8100290 2023-11-16T23:58:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.883+0000 7f61dd59a700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 0x7f61c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:52.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.883+0000 7f61dd59a700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 0x7f61c40a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61c800b2e0 tx=0x7f61c8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:52.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:52.885+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f61d4096800 con 0x7f61d8100290 2023-11-16T23:58:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:52 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2146560557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:52 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/35293516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:52 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1941850311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:52 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2146560557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:52 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/35293516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:52 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1941850311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:52 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2146560557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:52 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/35293516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:52 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1941850311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.140+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61d81a0f40 con 0x7f61d8100290 2023-11-16T23:58:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.141+0000 7f61ceffd700 1 -- 172.21.15.23:0/2561991894 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10753 (secure 0 0 0) 0x7f61d4003dd0 con 0x7f61d8100290 2023-11-16T23:58:53.143 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:53.143 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:58:51.234117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 msgr2=0x7f61c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 0x7f61c40a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61c800b2e0 tx=0x7f61c8005fb0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 msgr2=0x7f61d81981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e970 tx=0x7f61d400ec80 comp rx=0 tx=0).stop 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 shutdown_connections 2023-11-16T23:58:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f61c40a1150 0x7f61c40a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61d8101e10 0x7f61d819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d8100290 0x7f61d81981c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 --2- 172.21.15.23:0/2561991894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d8101480 0x7f61d8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 >> 172.21.15.23:0/2561991894 conn(0x7f61d80fb870 msgr2=0x7f61d80fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 shutdown_connections 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.144+0000 7f61dffff700 1 -- 172.21.15.23:0/2561991894 wait complete. 2023-11-16T23:58:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2023-11-16T23:58:53.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:58:53.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.368+0000 7fe0fbdff700 1 -- 172.21.15.23:0/706932485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101480 msgr2=0x7fe0f41018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.368+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/706932485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101480 0x7fe0f41018d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40098f0 tx=0x7fe0e4009c00 comp rx=0 tx=0).stop 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 -- 172.21.15.23:0/706932485 shutdown_connections 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/706932485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101e10 0x7fe0f41086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/706932485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 0x7fe0f41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/706932485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101480 0x7fe0f41018d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 -- 172.21.15.23:0/706932485 >> 172.21.15.23:0/706932485 conn(0x7fe0f40fb870 msgr2=0x7fe0f40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 -- 172.21.15.23:0/706932485 shutdown_connections 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.369+0000 7fe0fbdff700 1 -- 172.21.15.23:0/706932485 wait complete. 2023-11-16T23:58:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.370+0000 7fe0fbdff700 1 Processor -- start 2023-11-16T23:58:53.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.370+0000 7fe0fbdff700 1 -- start start 2023-11-16T23:58:53.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 0x7fe0f4195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101e10 0x7fe0f419d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419db20 con 0x7fe0f4101480 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419dc60 con 0x7fe0f4100290 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fbdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f419ddf0 con 0x7fe0f4101e10 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56474/0 (socket says 172.21.15.23:56474) 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 -- 172.21.15.23:0/3301976272 learned_addr learned my addr 172.21.15.23:0/3301976272 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f9b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 0x7fe0f4195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0fa39c700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101e10 0x7fe0f419d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 -- 172.21.15.23:0/3301976272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101e10 msgr2=0x7fe0f419d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101e10 0x7fe0f419d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 -- 172.21.15.23:0/3301976272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 msgr2=0x7fe0f4195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.371+0000 7fe0f939a700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 0x7fe0f4195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0f939a700 1 -- 172.21.15.23:0/3301976272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec009d00 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0f939a700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009fd0 tx=0x7fe0e400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e401c070 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0e400f7d0 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e4009520 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4004ad0 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.372+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f410bbf0 con 0x7fe0f4101480 2023-11-16T23:58:53.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.373+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f410bd80 con 0x7fe0f4101480 2023-11-16T23:58:53.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.375+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe0e4020070 con 0x7fe0f4101480 2023-11-16T23:58:53.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.376+0000 7fe0eaffd700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 0x7fe0e00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.377+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe0e40cd750 con 0x7fe0f4101480 2023-11-16T23:58:53.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.377+0000 7fe0f9b9b700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 0x7fe0e00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.377+0000 7fe0f9b9b700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 0x7fe0e00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe0f41012e0 tx=0x7fe0f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.381+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe0e405c730 con 0x7fe0f4101480 2023-11-16T23:58:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.591+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0f4107870 con 0x7fe0f4101480 2023-11-16T23:58:53.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.593+0000 7fe0eaffd700 1 -- 172.21.15.23:0/3301976272 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fe0e4096d40 con 0x7fe0f4101480 2023-11-16T23:58:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:58:53.594 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:58:53.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 msgr2=0x7fe0e00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 0x7fe0e00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe0f41012e0 tx=0x7fe0f0009260 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 msgr2=0x7fe0f41964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009fd0 tx=0x7fe0e400b790 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 shutdown_connections 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe0e00a1150 0x7fe0e00a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0f4101e10 0x7fe0f419d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0f4100290 0x7fe0f4195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 --2- 172.21.15.23:0/3301976272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0f4101480 0x7fe0f41964e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 >> 172.21.15.23:0/3301976272 conn(0x7fe0f40fb870 msgr2=0x7fe0f41049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 shutdown_connections 2023-11-16T23:58:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.595+0000 7fe0fbdff700 1 -- 172.21.15.23:0/3301976272 wait complete. 2023-11-16T23:58:53.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-16T23:58:46.253225+0000', 'modified': '2023-11-16T23:58:51.234117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6839/1637342449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1637342449}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1637342449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-16T23:58:53.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T23:58:53.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T23:58:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 -- 172.21.15.23:0/1786657839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 msgr2=0x7f549c101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 --2- 172.21.15.23:0/1786657839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c101870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f548c009990 tx=0x7f548c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 -- 172.21.15.23:0/1786657839 shutdown_connections 2023-11-16T23:58:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 --2- 172.21.15.23:0/1786657839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 0x7f549c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 --2- 172.21.15.23:0/1786657839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 --2- 172.21.15.23:0/1786657839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c101870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 -- 172.21.15.23:0/1786657839 >> 172.21.15.23:0/1786657839 conn(0x7f549c0fb810 msgr2=0x7f549c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 -- 172.21.15.23:0/1786657839 shutdown_connections 2023-11-16T23:58:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.821+0000 7f54a47ff700 1 -- 172.21.15.23:0/1786657839 wait complete. 2023-11-16T23:58:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.822+0000 7f54a47ff700 1 Processor -- start 2023-11-16T23:58:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.822+0000 7f54a47ff700 1 -- start start 2023-11-16T23:58:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 0x7f549c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c073bc0 con 0x7f549c101420 2023-11-16T23:58:53.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c073d30 con 0x7f549c101db0 2023-11-16T23:58:53.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a47ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c1a0020 con 0x7f549c100230 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a259b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56492/0 (socket says 172.21.15.23:56492) 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 -- 172.21.15.23:0/3293861071 learned_addr learned my addr 172.21.15.23:0/3293861071 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 0x7f549c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 -- 172.21.15.23:0/3293861071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 msgr2=0x7f549c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 -- 172.21.15.23:0/3293861071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 msgr2=0x7f549c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 0x7f549c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a1d9a700 1 -- 172.21.15.23:0/3293861071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c0095f0 con 0x7f549c101420 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.823+0000 7f54a259b700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f548c004760 tx=0x7f548c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54937fe700 1 -- 172.21.15.23:0/3293861071 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c01d070 con 0x7f549c101420 2023-11-16T23:58:53.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c10bbf0 con 0x7f549c101420 2023-11-16T23:58:53.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54937fe700 1 -- 172.21.15.23:0/3293861071 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f548c004830 con 0x7f549c101420 2023-11-16T23:58:53.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54937fe700 1 -- 172.21.15.23:0/3293861071 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c005480 con 0x7f549c101420 2023-11-16T23:58:53.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.824+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c10ba60 con 0x7f549c101420 2023-11-16T23:58:53.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.829+0000 7f54937fe700 1 -- 172.21.15.23:0/3293861071 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f548c00fad0 con 0x7f549c101420 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 Processor -- start 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 -- start start 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adefbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adefcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefd0c0 0x5627adefd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ade01010 con 0x5627adefd0c0 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ade53850 con 0x5627adefbbb0 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54b8141b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ade4cce0 con 0x5627adefc500 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54937fe700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 0x7f54880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54a2d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adefbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54a259b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adefcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54a259b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adefcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52000/0 (socket says 172.21.15.23:52000) 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.830+0000 7f54a259b700 1 -- 172.21.15.23:0/3667011494 learned_addr learned my addr 172.21.15.23:0/3667011494 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54937fe700 1 -- 172.21.15.23:0/3293861071 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f548c0cdcb0 con 0x7f549c101420 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefd0c0 0x5627adefd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 0x7f54880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 -- 172.21.15.23:0/3667011494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 msgr2=0x5627adefcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adefcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 -- 172.21.15.23:0/3667011494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefd0c0 msgr2=0x5627adefd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefd0c0 0x5627adefd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 -- 172.21.15.23:0/3667011494 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627adcce5e0 con 0x5627adefbbb0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adefbfc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5494014b20 tx=0x7f5494014e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c39ced2fd8e905 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 0x7f54880a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5494009000 tx=0x7f549401c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.831+0000 7f5490ff9700 1 -- 172.21.15.23:0/3667011494 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549400e040 con 0x5627adefbbb0 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f5490ff9700 1 -- 172.21.15.23:0/3667011494 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5494017eb0 con 0x5627adefbbb0 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f5490ff9700 1 -- 172.21.15.23:0/3667011494 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494019830 con 0x5627adefbbb0 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 -- 172.21.15.23:0/3667011494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 msgr2=0x5627adefbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adefbfc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5494014b20 tx=0x7f5494014e30 comp rx=0 tx=0).stop 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 -- 172.21.15.23:0/3667011494 shutdown_connections 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefd0c0 0x5627adefd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adefbfc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 --2- 172.21.15.23:0/3667011494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adefcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 -- 172.21.15.23:0/3667011494 >> 172.21.15.23:0/3667011494 conn(0x5627adef1310 msgr2=0x5627adef6620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.832+0000 7f54b8141b80 1 -- 172.21.15.23:0/3667011494 shutdown_connections 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.833+0000 7f54b8141b80 1 -- 172.21.15.23:0/3667011494 wait complete. 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.833+0000 7f54b8141b80 1 -- start start 2023-11-16T23:58:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.833+0000 7f54b8141b80 1 Processor -- start 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adf05b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefd0c0 0x5627adf0cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627addee890 con 0x5627adefbbb0 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627addee530 con 0x5627adefd0c0 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627adcd08d0 con 0x5627adefc500 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54b8141b80 10 client.0 _ll_register_callbacks cb 0x5627ade0c730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56502/0 (socket says 172.21.15.23:56502) 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 -- 172.21.15.23:0/798686919 learned_addr learned my addr 172.21.15.23:0/798686919 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adf05b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefd0c0 0x5627adf0cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 -- 172.21.15.23:0/798686919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 msgr2=0x5627adf05b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:53.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adf05b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 -- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefd0c0 msgr2=0x5627adf0cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:53.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefd0c0 0x5627adf0cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.834+0000 7f54a259b700 1 -- 172.21.15.23:0/798686919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5494014700 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f54a259b700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5498011ce0 tx=0x7f5498009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6af6a7f32372200 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498016070 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f549800fbc0 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627adeeead0 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f549800fdc0 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f54b8141b80 10 client.14655 fetch_fsmap learned FSMap version 17 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5627ade13140 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5627adeeead0 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.835+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f5498019390 con 0x5627adefbbb0 2023-11-16T23:58:53.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 10 client.14655 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 10 client.14655 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 20 client.14655 populate_metadata read hostname 'smithi023' 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf10920 0x5627adf0fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 4 client.14655 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627ade0f050 con 0x5627adf10920 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf10920 0x5627adf0fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f5485ffb700 10 client.14655 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.836+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf10920 0x5627adf0fcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a289a4d36b2451e7 server_cookie=87e5f831f67ef3b in_seq=0 out_seq=0 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.837+0000 7f5485ffb700 1 -- 172.21.15.23:0/798686919 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5627ade0f050 con 0x5627adf10920 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.837+0000 7f5485ffb700 10 client.14655 handle_command_reply: tid=0 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.837+0000 7f54b8141b80 1 client.14655 shutdown 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.837+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 msgr2=0x5627adf055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:53.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5498011ce0 tx=0x7f5498009ec0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 shutdown_connections 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefd0c0 0x5627adf0cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf10920 0x5627adf0fcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf055f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 --2- 172.21.15.23:0/798686919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefc500 0x5627adf05b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f5485ffb700 0 client.14655 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 >> 172.21.15.23:0/798686919 conn(0x5627adef1310 msgr2=0x5627adef5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 shutdown_connections 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 1 -- 172.21.15.23:0/798686919 wait complete. 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.838+0000 7f54b8141b80 20 client.14655 trim_cache size 0 max 16384 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 Processor -- start 2023-11-16T23:58:53.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 -- start start 2023-11-16T23:58:53.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefc500 0x5627adf0cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf0fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627adf067e0 con 0x5627adefbbb0 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627addf6730 con 0x5627adefc500 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54b8141b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627add58ed0 con 0x5627adefd0c0 2023-11-16T23:58:54.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54a259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54a259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56506/0 (socket says 172.21.15.23:56506) 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54a259b700 1 -- 172.21.15.23:0/2642095256 learned_addr learned my addr 172.21.15.23:0/2642095256 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefc500 0x5627adf0cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.857+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf0fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 -- 172.21.15.23:0/2642095256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 msgr2=0x5627adf0fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf0fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 -- 172.21.15.23:0/2642095256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefc500 msgr2=0x5627adf0cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefc500 0x5627adf0cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 -- 172.21.15.23:0/2642095256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627ade12e90 con 0x5627adefbbb0 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a2d9c700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf0fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54a259b700 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5498009340 tx=0x7f5498009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f458a237fca558f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54917fa700 1 -- 172.21.15.23:0/2642095256 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498016040 con 0x5627adefbbb0 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54917fa700 1 -- 172.21.15.23:0/2642095256 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5498019df0 con 0x5627adefbbb0 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.858+0000 7f54917fa700 1 -- 172.21.15.23:0/2642095256 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498003680 con 0x5627adefbbb0 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- 172.21.15.23:0/2642095256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 msgr2=0x5627adf04270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5498009340 tx=0x7f5498009f30 comp rx=0 tx=0).stop 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- 172.21.15.23:0/2642095256 shutdown_connections 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf0fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefbbb0 0x5627adf04270 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 --2- 172.21.15.23:0/2642095256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefc500 0x5627adf0cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- 172.21.15.23:0/2642095256 >> 172.21.15.23:0/2642095256 conn(0x5627adef1310 msgr2=0x5627adf19690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- 172.21.15.23:0/2642095256 shutdown_connections 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- 172.21.15.23:0/2642095256 wait complete. 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 -- start start 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.859+0000 7f54b8141b80 1 Processor -- start 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adf09620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf647c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627adf0cb80 con 0x5627adefc500 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627adf0ccf0 con 0x5627adefbbb0 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627adf0ceb0 con 0x5627adefd0c0 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54b8141b80 10 client.0 _ll_register_callbacks cb 0x7f54980117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T23:58:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a2d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf647c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56516/0 (socket says 172.21.15.23:56516) 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a2d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf647c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52036/0 (socket says 172.21.15.23:52036) 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a1d9a700 1 -- 172.21.15.23:0/1291441450 learned_addr learned my addr 172.21.15.23:0/1291441450 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.860+0000 7f54a259b700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adf09620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 -- 172.21.15.23:0/1291441450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 msgr2=0x5627adf647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf647c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 -- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 msgr2=0x5627adf09620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T23:58:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adf09620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 -- 172.21.15.23:0/1291441450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627ade13140 con 0x5627adefc500 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54a1d9a700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f548c097460 tx=0x7f548c09abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb68ea28fc2a0bfd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f5485ffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c09ef90 con 0x5627adefc500 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f548c0a3320 con 0x5627adefc500 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5494017eb0 con 0x5627adefc500 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f548c00fd80 con 0x5627adefc500 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54b8141b80 10 client.14661 fetch_fsmap learned FSMap version 17 2023-11-16T23:58:55.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.861+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5627adc63c10 con 0x5627adefc500 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5494017eb0 con 0x5627adefc500 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f548c09b250 con 0x5627adefc500 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 10 client.14661 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 10 client.14661 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 20 client.14661 populate_metadata read hostname 'smithi023' 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf69470 0x5627adf65de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 4 client.14661 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5627ade1b610 con 0x5627adf69470 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.862+0000 7f54a259b700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf69470 0x5627adf65de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f5485ffb700 10 client.14661 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54a259b700 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf69470 0x5627adf65de0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b431b64f8c5e1 server_cookie=f8b6c8376d4e17cc in_seq=0 out_seq=0 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f5485ffb700 1 -- 172.21.15.23:0/1291441450 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5627ade1b610 con 0x5627adf69470 2023-11-16T23:58:55.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f5485ffb700 10 client.14661 handle_command_reply: tid=0 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 client.14661 shutdown 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 msgr2=0x5627adf09b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f548c097460 tx=0x7f548c09abc0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 shutdown_connections 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5627adefd0c0 0x5627adf647c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627adefbbb0 0x5627adf09620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5627adf69470 0x5627adf65de0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 --2- 172.21.15.23:0/1291441450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627adefc500 0x5627adf09b60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f5485ffb700 0 client.14661 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 >> 172.21.15.23:0/1291441450 conn(0x5627adef1310 msgr2=0x5627adf0fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 shutdown_connections 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.863+0000 7f54b8141b80 1 -- 172.21.15.23:0/1291441450 wait complete. 2023-11-16T23:58:55.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54b8141b80 20 client.14661 trim_cache size 0 max 16384 2023-11-16T23:58:55.811 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24572, 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 7.6040937150000003, 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 17, 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2023-11-16T23:58:56.001 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 87, 2023-11-16T23:58:56.002 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 12.802891031 2023-11-16T23:58:56.002 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 msgr2=0x7f54880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 0x7f54880a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5494009000 tx=0x7f549401c000 comp rx=0 tx=0).stop 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 msgr2=0x7f549c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f548c004760 tx=0x7f548c004790 comp rx=0 tx=0).stop 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 shutdown_connections 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f549c101db0 0x7f549c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f54880a1150 0x7f54880a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f549c100230 0x7f549c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 --2- 172.21.15.23:0/3293861071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c101420 0x7f549c072ee0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:58:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 >> 172.21.15.23:0/3293861071 conn(0x7f549c0fb810 msgr2=0x7f549c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:58:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 shutdown_connections 2023-11-16T23:58:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:58:53.864+0000 7f54a47ff700 1 -- 172.21.15.23:0/3293861071 wait complete. 2023-11-16T23:58:56.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:53 smithi023 ceph-mon[97824]: pgmap v299: 129 pgs: 3 creating+peering, 113 active+clean, 13 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s wr, 6 op/s 2023-11-16T23:58:56.575 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:53 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2561991894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.575 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:53 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3301976272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.575 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:53 smithi156 ceph-mon[131105]: pgmap v299: 129 pgs: 3 creating+peering, 113 active+clean, 13 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s wr, 6 op/s 2023-11-16T23:58:56.575 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:53 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2561991894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.576 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:53 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3301976272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:53 smithi173 ceph-mon[131534]: pgmap v299: 129 pgs: 3 creating+peering, 113 active+clean, 13 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s wr, 6 op/s 2023-11-16T23:58:56.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:53 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2561991894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:53 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3301976272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:58:56.579 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T23:59:10.144 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:10.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T23:59:10.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:56 smithi023 ceph-mon[97824]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2023-11-16T23:59:10.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:58 smithi023 ceph-mon[97824]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:58:59 smithi023 ceph-mon[97824]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:02 smithi023 ceph-mon[97824]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:04 smithi023 ceph-mon[97824]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:06 smithi023 ceph-mon[97824]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:08 smithi023 ceph-mon[97824]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:09 smithi023 ceph-mon[97824]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:56 smithi156 ceph-mon[131105]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:58 smithi156 ceph-mon[131105]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:58:59 smithi156 ceph-mon[131105]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:02 smithi156 ceph-mon[131105]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:04 smithi156 ceph-mon[131105]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:06 smithi156 ceph-mon[131105]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:08 smithi156 ceph-mon[131105]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:09 smithi156 ceph-mon[131105]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:56 smithi173 ceph-mon[131534]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 4 op/s 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:58 smithi173 ceph-mon[131534]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:58:59 smithi173 ceph-mon[131534]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:02 smithi173 ceph-mon[131534]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:04 smithi173 ceph-mon[131534]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:06 smithi173 ceph-mon[131534]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:08 smithi173 ceph-mon[131534]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:10.301 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:09 smithi173 ceph-mon[131534]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:12.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:12 smithi023 ceph-mon[97824]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/932061975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 msgr2=0x7f7b44104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/932061975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44104eb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b300099c0 tx=0x7f7b30009cd0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/932061975 shutdown_connections 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/932061975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b441055e0 0x7f7b4410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/932061975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b44100540 0x7f7b44100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/932061975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44104eb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/932061975 >> 172.21.15.23:0/932061975 conn(0x7f7b440fbdc0 msgr2=0x7f7b440fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/932061975 shutdown_connections 2023-11-16T23:59:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.570+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/932061975 wait complete. 2023-11-16T23:59:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.571+0000 7f7b4ab5a700 1 Processor -- start 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.571+0000 7f7b4ab5a700 1 -- start start 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b44100540 0x7f7b44072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b441055e0 0x7f7b440738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a4840 con 0x7f7b44100ee0 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a49b0 con 0x7f7b44100540 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b4ab5a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441a4b70 con 0x7f7b441055e0 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53762/0 (socket says 172.21.15.23:53762) 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 -- 172.21.15.23:0/745476400 learned_addr learned my addr 172.21.15.23:0/745476400 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b488f6700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b44100540 0x7f7b44072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b490f7700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b441055e0 0x7f7b440738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 -- 172.21.15.23:0/745476400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b441055e0 msgr2=0x7f7b440738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b441055e0 0x7f7b440738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.572+0000 7f7b43fff700 1 -- 172.21.15.23:0/745476400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b44100540 msgr2=0x7f7b44072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b43fff700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b44100540 0x7f7b44072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b43fff700 1 -- 172.21.15.23:0/745476400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300095f0 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b43fff700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009990 tx=0x7f7b3000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3001d070 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b30005030 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b44110360 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30004a50 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.573+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441101d0 con 0x7f7b44100ee0 2023-11-16T23:59:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.574+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b441104f0 con 0x7f7b44100ee0 2023-11-16T23:59:12.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.576+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7b30005430 con 0x7f7b44100ee0 2023-11-16T23:59:12.580 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.578+0000 7f7b41ffb700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 0x7f7b340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:12.580 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.579+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7b300cd820 con 0x7f7b44100ee0 2023-11-16T23:59:12.580 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.579+0000 7f7b488f6700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 0x7f7b340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:12.580 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.579+0000 7f7b488f6700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 0x7f7b340a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009c60 tx=0x7f7b38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:12.584 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.583+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7b30096db0 con 0x7f7b44100ee0 2023-11-16T23:59:12.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:12 smithi173 ceph-mon[131534]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:12 smithi156 ceph-mon[131105]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:12.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.842+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b44066ea0 con 0x7f7b44100ee0 2023-11-16T23:59:12.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.842+0000 7f7b41ffb700 1 -- 172.21.15.23:0/745476400 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7b3009a3d0 con 0x7f7b44100ee0 2023-11-16T23:59:12.844 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T23:59:12.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 msgr2=0x7f7b340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:12.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 0x7f7b340a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009c60 tx=0x7f7b38009260 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 msgr2=0x7f7b44073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009990 tx=0x7f7b3000f690 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 shutdown_connections 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b340a1150 0x7f7b340a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b441055e0 0x7f7b440738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b44100540 0x7f7b44072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 --2- 172.21.15.23:0/745476400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b44100ee0 0x7f7b44073330 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 >> 172.21.15.23:0/745476400 conn(0x7f7b440fbdc0 msgr2=0x7f7b440fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 shutdown_connections 2023-11-16T23:59:12.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:12.845+0000 7f7b4ab5a700 1 -- 172.21.15.23:0/745476400 wait complete. 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T23:59:13.346 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T23:59:13.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:13.347 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2023-11-16T23:59:13.393 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41636sec preferred_lft 41636sec 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T23:59:13.394 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2023-11-16T23:59:13.395 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2023-11-16T23:59:13.396 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T23:59:13.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T23:59:13.396 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-16T23:59:13.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:13 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/745476400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T23:59:13.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:13 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/745476400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T23:59:13.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:13 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/745476400' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T23:59:13.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:13.878 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T23:59:13.950 INFO:teuthology.orchestra.run.smithi023.stdout:1 2023-11-16T23:59:13.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:13.957 DEBUG:teuthology.orchestra.run.smithi023:> route 2023-11-16T23:59:14.027 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2023-11-16T23:59:14.028 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T23:59:14.028 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T23:59:14.028 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T23:59:14.028 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T23:59:14.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T23:59:14.031 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-16T23:59:14.390 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:14 smithi023 ceph-mon[97824]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:14 smithi173 ceph-mon[131534]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:14 smithi156 ceph-mon[131105]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:14.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:14.713 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-11-16T23:59:14.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:14.761 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2023-11-16T23:59:14.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:14.806 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-16T23:59:14.806 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-16T23:59:14.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T23:59:14.806 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-16T23:59:14.806 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-16T23:59:14.993 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-16T23:59:14.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-16T23:59:14.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-16T23:59:14.995 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-16T23:59:15.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:15.280 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-16T23:59:15.280 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-16T23:59:15.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2023-11-16T23:59:15.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T23:59:15.280 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-16T23:59:15.459 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-16T23:59:15.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T23:59:15.459 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:15.504 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T23:59:15.504 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T23:59:15.505 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:15.589 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-11-16T23:59:15.702 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T23:59:15.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T23:59:15.773 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T23:59:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T23:59:15.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:15.776 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-11-16T23:59:15.820 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-16T23:59:15.821 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-11-16T23:59:15.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-11-16T23:59:15.938 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T23:59:15.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T23:59:16.009 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T23:59:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T23:59:16.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:16.014 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-11-16T23:59:16.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2023-11-16T23:59:16.063+0000 7f3e974815c0 -1 init, newargv = 0x55b95903d520 newargc=15 2023-11-16T23:59:16.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[134396]: starting ceph client 2023-11-16T23:59:16.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[134396]: starting fuse 2023-11-16T23:59:16.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:16 smithi023 ceph-mon[97824]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:16 smithi173 ceph-mon[131534]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:16 smithi156 ceph-mon[131105]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:17.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-11-16T23:59:17.239 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T23:59:17.240 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T23:59:17.311 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T23:59:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T23:59:17.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:17.314 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-11-16T23:59:17.358 INFO:teuthology.orchestra.run.smithi023.stdout:121 2023-11-16T23:59:17.358 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-11-16T23:59:17.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> import re 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> import os 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2023-11-16T23:59:17.359 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> return f 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2023-11-16T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi023:> ' 2023-11-16T23:59:17.545 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.134396.asok 2023-11-16T23:59:17.723 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.134396.asok 2023-11-16T23:59:17.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:17.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.134396.asok status 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "1c9565146c8647d4e00eb6c7ae626dc192680e7b", 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev)", 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2023-11-16T23:59:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "134396", 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14673, 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 14673 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-16T23:59:17.956 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1750302468 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1750302468 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.14673 172.21.15.23:0/1750302468", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "172.21.15.23:0/1750302468", 2023-11-16T23:59:17.957 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 17, 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 88, 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2023-11-16T23:59:17.958 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-16T23:59:17.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:17.970 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2023-11-16T23:59:17.970 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:18.016 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2023-11-16T23:59:18.016 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:18.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:18.016 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:18.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:18.129 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-16T23:59:18.256 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2023-11-16T23:59:18.256 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-16T23:59:18.256 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:18.257 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:59:18.258 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T23:59:18.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:59:18.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:18 smithi023 ceph-mon[97824]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:18.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 -- 172.21.15.23:0/3552373585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 msgr2=0x7fe61810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:18.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/3552373585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe61810ec40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe61000b0a0 tx=0x7fe61000b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 -- 172.21.15.23:0/3552373585 shutdown_connections 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/3552373585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe61810ec40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/3552373585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 0x7fe618072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/3552373585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 0x7fe6180739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.549+0000 7fe61e3ff700 1 -- 172.21.15.23:0/3552373585 >> 172.21.15.23:0/3552373585 conn(0x7fe6180fb870 msgr2=0x7fe6180fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.550+0000 7fe61e3ff700 1 -- 172.21.15.23:0/3552373585 shutdown_connections 2023-11-16T23:59:18.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.550+0000 7fe61e3ff700 1 -- 172.21.15.23:0/3552373585 wait complete. 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 Processor -- start 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 -- start start 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 0x7fe61819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 0x7fe61819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a4230 con 0x7fe61810c860 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a4370 con 0x7fe618074b30 2023-11-16T23:59:18.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61e3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a4500 con 0x7fe618073550 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61c99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61c99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53778/0 (socket says 172.21.15.23:53778) 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe61c99c700 1 -- 172.21.15.23:0/394678277 learned_addr learned my addr 172.21.15.23:0/394678277 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.551+0000 7fe617fff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 0x7fe61819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe6177fe700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 0x7fe61819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 -- 172.21.15.23:0/394678277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 msgr2=0x7fe61819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 0x7fe61819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 -- 172.21.15.23:0/394678277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 msgr2=0x7fe61819cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 0x7fe61819cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 -- 172.21.15.23:0/394678277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610009d00 con 0x7fe61810c860 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61c99c700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe610009fd0 tx=0x7fe61000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000e040 con 0x7fe61810c860 2023-11-16T23:59:18.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6100094a0 con 0x7fe61810c860 2023-11-16T23:59:18.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe618112270 con 0x7fe61810c860 2023-11-16T23:59:18.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.552+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61001b6c0 con 0x7fe61810c860 2023-11-16T23:59:18.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.553+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6181120e0 con 0x7fe61810c860 2023-11-16T23:59:18.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.553+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6181023b0 con 0x7fe61810c860 2023-11-16T23:59:18.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.556+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe61001b820 con 0x7fe61810c860 2023-11-16T23:59:18.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.557+0000 7fe6157fa700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 0x7fe6000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:18.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.557+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe6100ce3e0 con 0x7fe61810c860 2023-11-16T23:59:18.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.557+0000 7fe617fff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 0x7fe6000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:18.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.558+0000 7fe617fff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 0x7fe6000a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe618074f00 tx=0x7fe608009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:18.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.560+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe610097970 con 0x7fe61810c860 2023-11-16T23:59:18.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:18 smithi156 ceph-mon[131105]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:18.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:18 smithi173 ceph-mon[131534]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:18.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.768+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe618067a60 con 0x7fe61810c860 2023-11-16T23:59:18.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.770+0000 7fe6157fa700 1 -- 172.21.15.23:0/394678277 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fe61009af90 con 0x7fe61810c860 2023-11-16T23:59:18.772 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:18.772 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:18.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 msgr2=0x7fe6000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:18.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 0x7fe6000a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe618074f00 tx=0x7fe608009260 comp rx=0 tx=0).stop 2023-11-16T23:59:18.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 msgr2=0x7fe6181a3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe610009fd0 tx=0x7fe61000bce0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 shutdown_connections 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61810c860 0x7fe6181a3c60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe6000a1150 0x7fe6000a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe618073550 0x7fe61819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 --2- 172.21.15.23:0/394678277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe618074b30 0x7fe61819cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 >> 172.21.15.23:0/394678277 conn(0x7fe6180fb870 msgr2=0x7fe6180fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 shutdown_connections 2023-11-16T23:59:18.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:18.773+0000 7fe61e3ff700 1 -- 172.21.15.23:0/394678277 wait complete. 2023-11-16T23:59:18.793 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T23:59:18.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T23:59:19.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.000+0000 7f11267ff700 1 -- 172.21.15.23:0/846337329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1120073550 msgr2=0x7f11200739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.000+0000 7f11267ff700 1 --2- 172.21.15.23:0/846337329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1120073550 0x7f11200739c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1114009990 tx=0x7f1114009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 -- 172.21.15.23:0/846337329 shutdown_connections 2023-11-16T23:59:19.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 --2- 172.21.15.23:0/846337329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f112010c890 0x7f112010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 --2- 172.21.15.23:0/846337329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f1120072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 --2- 172.21.15.23:0/846337329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1120073550 0x7f11200739c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 -- 172.21.15.23:0/846337329 >> 172.21.15.23:0/846337329 conn(0x7f11200fb8a0 msgr2=0x7f11200fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 -- 172.21.15.23:0/846337329 shutdown_connections 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.001+0000 7f11267ff700 1 -- 172.21.15.23:0/846337329 wait complete. 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 Processor -- start 2023-11-16T23:59:19.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 -- start start 2023-11-16T23:59:19.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f112019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11201a4220 con 0x7f112010c890 2023-11-16T23:59:19.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11201a4360 con 0x7f1120073550 2023-11-16T23:59:19.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f11267ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11201a44f0 con 0x7f1120074b30 2023-11-16T23:59:19.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f1124d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f111ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f111ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:39148/0 (socket says 172.21.15.23:39148) 2023-11-16T23:59:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.002+0000 7f1124d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53792/0 (socket says 172.21.15.23:53792) 2023-11-16T23:59:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111ffff700 1 -- 172.21.15.23:0/1258971595 learned_addr learned my addr 172.21.15.23:0/1258971595 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f112019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 -- 172.21.15.23:0/1258971595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 msgr2=0x7f112019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 -- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 msgr2=0x7f11201a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 -- 172.21.15.23:0/1258971595 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11140095f0 con 0x7f1120074b30 2023-11-16T23:59:19.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f1124d9c700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:19.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111ffff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:19.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111f7fe700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f112019cbe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f111400b680 tx=0x7f1114005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111401d070 con 0x7f1120074b30 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.003+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11201122a0 con 0x7f1120074b30 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.004+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1114021410 con 0x7f1120074b30 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.004+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114017650 con 0x7f1120074b30 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.004+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1120112110 con 0x7f1120074b30 2023-11-16T23:59:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.005+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11201023e0 con 0x7f1120074b30 2023-11-16T23:59:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.007+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f11140217f0 con 0x7f1120074b30 2023-11-16T23:59:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.008+0000 7f111d7fa700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 0x7f11080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.008+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f11140cdfe0 con 0x7f1120074b30 2023-11-16T23:59:19.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.008+0000 7f111ffff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 0x7f11080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.009+0000 7f111ffff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 0x7f11080a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1110005950 tx=0x7f1110009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.010+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1114097690 con 0x7f1120074b30 2023-11-16T23:59:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.227+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1120067b00 con 0x7f1120074b30 2023-11-16T23:59:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.228+0000 7f111d7fa700 1 -- 172.21.15.23:0/1258971595 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11140260a0 con 0x7f1120074b30 2023-11-16T23:59:19.229 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-16T23:59:19.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 msgr2=0x7f11080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 0x7f11080a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1110005950 tx=0x7f1110009260 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 msgr2=0x7f112019cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f112019cbe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f111400b680 tx=0x7f1114005740 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 shutdown_connections 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112010c890 0x7f11201a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1120073550 0x7f112019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f11080a11d0 0x7f11080a3680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 --2- 172.21.15.23:0/1258971595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1120074b30 0x7f112019cbe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 >> 172.21.15.23:0/1258971595 conn(0x7f11200fb8a0 msgr2=0x7f11200fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 shutdown_connections 2023-11-16T23:59:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.230+0000 7f11267ff700 1 -- 172.21.15.23:0/1258971595 wait complete. 2023-11-16T23:59:19.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 -- 172.21.15.23:0/3719425264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7378101e40 msgr2=0x7f7378108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 --2- 172.21.15.23:0/3719425264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7378101e40 0x7f7378108700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f737000b0a0 tx=0x7f737000b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 -- 172.21.15.23:0/3719425264 shutdown_connections 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 --2- 172.21.15.23:0/3719425264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7378101e40 0x7f7378108700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 --2- 172.21.15.23:0/3719425264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73781002c0 0x7f73781006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 --2- 172.21.15.23:0/3719425264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781014b0 0x7f7378101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 -- 172.21.15.23:0/3719425264 >> 172.21.15.23:0/3719425264 conn(0x7f73780fb8a0 msgr2=0x7f73780fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 -- 172.21.15.23:0/3719425264 shutdown_connections 2023-11-16T23:59:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.458+0000 7f737dfff700 1 -- 172.21.15.23:0/3719425264 wait complete. 2023-11-16T23:59:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 Processor -- start 2023-11-16T23:59:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 -- start start 2023-11-16T23:59:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 0x7f7378198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378101e40 0x7f737819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.459+0000 7f737dfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737819feb0 con 0x7f73781014b0 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f737dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737819fff0 con 0x7f7378101e40 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f737dfff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73781a0180 con 0x7f73781002c0 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53816/0 (socket says 172.21.15.23:53816) 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 -- 172.21.15.23:0/2019115552 learned_addr learned my addr 172.21.15.23:0/2019115552 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f73777fe700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 0x7f7378198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f7377fff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378101e40 0x7f737819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 -- 172.21.15.23:0/2019115552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 msgr2=0x7f7378198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 0x7f7378198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 -- 172.21.15.23:0/2019115552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378101e40 msgr2=0x7f737819f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378101e40 0x7f737819f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 -- 172.21.15.23:0/2019115552 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009d00 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f73777fe700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 0x7f7378198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.460+0000 7f736ffff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9a0 tx=0x7f736000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.461+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360009790 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.461+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736000cbd0 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.461+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737810bc90 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.461+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360018680 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.461+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737810bb00 con 0x7f73781014b0 2023-11-16T23:59:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.462+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73781a0870 con 0x7f73781014b0 2023-11-16T23:59:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.464+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f736000cd40 con 0x7f73781014b0 2023-11-16T23:59:19.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.465+0000 7f73757fa700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 0x7f73640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.466+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7360014070 con 0x7f73781014b0 2023-11-16T23:59:19.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.466+0000 7f73777fe700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 0x7f73640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.466+0000 7f73777fe700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 0x7f73640a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7378101310 tx=0x7f7368009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.470+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7360051730 con 0x7f73781014b0 2023-11-16T23:59:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.682+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73781a0e20 con 0x7f73781014b0 2023-11-16T23:59:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.683+0000 7f73757fa700 1 -- 172.21.15.23:0/2019115552 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73600962b0 con 0x7f73781014b0 2023-11-16T23:59:19.684 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 msgr2=0x7f73640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 0x7f73640a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7378101310 tx=0x7f7368009260 comp rx=0 tx=0).stop 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 msgr2=0x7f7378198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9a0 tx=0x7f736000ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 shutdown_connections 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378101e40 0x7f737819f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73640a1150 0x7f73640a3600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73781002c0 0x7f7378198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 --2- 172.21.15.23:0/2019115552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73781014b0 0x7f7378198870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 >> 172.21.15.23:0/2019115552 conn(0x7f73780fb8a0 msgr2=0x7f73780fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 shutdown_connections 2023-11-16T23:59:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.685+0000 7f737dfff700 1 -- 172.21.15.23:0/2019115552 wait complete. 2023-11-16T23:59:19.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T23:59:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 -- 172.21.15.23:0/2326177941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 msgr2=0x7f2528108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 --2- 172.21.15.23:0/2326177941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f2528108710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f252000b0a0 tx=0x7f252000b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 -- 172.21.15.23:0/2326177941 shutdown_connections 2023-11-16T23:59:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 --2- 172.21.15.23:0/2326177941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f2528108710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 --2- 172.21.15.23:0/2326177941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281002d0 0x7f25281006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 --2- 172.21.15.23:0/2326177941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281014c0 0x7f2528101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 -- 172.21.15.23:0/2326177941 >> 172.21.15.23:0/2326177941 conn(0x7f25280fb8b0 msgr2=0x7f25280fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 -- 172.21.15.23:0/2326177941 shutdown_connections 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.912+0000 7f252e3ff700 1 -- 172.21.15.23:0/2326177941 wait complete. 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 Processor -- start 2023-11-16T23:59:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 -- start start 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281002d0 0x7f2528196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 0x7f25281966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252819dd00 con 0x7f2528101e50 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252819de40 con 0x7f25281014c0 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.913+0000 7f252e3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252819dfd0 con 0x7f25281002d0 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53848/0 (socket says 172.21.15.23:53848) 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 -- 172.21.15.23:0/1127472151 learned_addr learned my addr 172.21.15.23:0/1127472151 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f2527fff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281002d0 0x7f2528196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f25277fe700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 0x7f25281966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 -- 172.21.15.23:0/1127472151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281002d0 msgr2=0x7f2528196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281002d0 0x7f2528196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 -- 172.21.15.23:0/1127472151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 msgr2=0x7f25281966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 0x7f25281966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 -- 172.21.15.23:0/1127472151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2520009d00 con 0x7f2528101e50 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f25277fe700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 0x7f25281966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.914+0000 7f252c99c700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f252000d010 tx=0x7f252000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.915+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252000e040 con 0x7f2528101e50 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.915+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2520003e10 con 0x7f2528101e50 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.915+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25200052a0 con 0x7f2528101e50 2023-11-16T23:59:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.915+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252810bca0 con 0x7f2528101e50 2023-11-16T23:59:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.915+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252810bb10 con 0x7f2528101e50 2023-11-16T23:59:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.916+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25281077f0 con 0x7f2528101e50 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.918+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2520005400 con 0x7f2528101e50 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.919+0000 7f25257fa700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 0x7f25100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.919+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f25200ce280 con 0x7f2528101e50 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.919+0000 7f2527fff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 0x7f25100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.920+0000 7f2527fff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 0x7f25100a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2528101320 tx=0x7f2518009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:19.921+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2520097890 con 0x7f2528101e50 2023-11-16T23:59:20.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/394678277' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:19 smithi023 ceph-mon[97824]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1258971595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2019115552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/394678277' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:20.114 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:19 smithi173 ceph-mon[131534]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:20.114 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1258971595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.114 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2019115552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/394678277' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:20.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:19 smithi156 ceph-mon[131105]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T23:59:20.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1258971595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2019115552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:20.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.133+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2528067b00 con 0x7f2528101e50 2023-11-16T23:59:20.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.134+0000 7f25257fa700 1 -- 172.21.15.23:0/1127472151 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f252009aeb0 con 0x7f2528101e50 2023-11-16T23:59:20.135 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 msgr2=0x7f25100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 0x7f25100a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2528101320 tx=0x7f2518009260 comp rx=0 tx=0).stop 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 msgr2=0x7f252819d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f252000d010 tx=0x7f252000bc80 comp rx=0 tx=0).stop 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 shutdown_connections 2023-11-16T23:59:20.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2528101e50 0x7f252819d730 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f25100a11d0 0x7f25100a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25281002d0 0x7f2528196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 --2- 172.21.15.23:0/1127472151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25281014c0 0x7f25281966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 >> 172.21.15.23:0/1127472151 conn(0x7f25280fb8b0 msgr2=0x7f2528104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 shutdown_connections 2023-11-16T23:59:20.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.136+0000 7f252e3ff700 1 -- 172.21.15.23:0/1127472151 wait complete. 2023-11-16T23:59:20.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.361+0000 7f1105bff700 1 -- 172.21.15.23:0/432834660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001002d0 msgr2=0x7f11001006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.361+0000 7f1105bff700 1 --2- 172.21.15.23:0/432834660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001002d0 0x7f11001006e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009990 tx=0x7f10f0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 -- 172.21.15.23:0/432834660 shutdown_connections 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 --2- 172.21.15.23:0/432834660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 0x7f1100108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 --2- 172.21.15.23:0/432834660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001002d0 0x7f11001006e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 --2- 172.21.15.23:0/432834660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001014c0 0x7f1100101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 -- 172.21.15.23:0/432834660 >> 172.21.15.23:0/432834660 conn(0x7f11000fb8b0 msgr2=0x7f11000fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 -- 172.21.15.23:0/432834660 shutdown_connections 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.362+0000 7f1105bff700 1 -- 172.21.15.23:0/432834660 wait complete. 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 Processor -- start 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 -- start start 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001002d0 0x7f1100198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 0x7f110019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110019fed0 con 0x7f11001014c0 2023-11-16T23:59:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001a0010 con 0x7f11001002d0 2023-11-16T23:59:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f1105bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001a01a0 con 0x7f1100101e50 2023-11-16T23:59:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f10feffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f10feffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53860/0 (socket says 172.21.15.23:53860) 2023-11-16T23:59:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f10feffd700 1 -- 172.21.15.23:0/2609766050 learned_addr learned my addr 172.21.15.23:0/2609766050 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f10ff7fe700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001002d0 0x7f1100198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.363+0000 7f10fffff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 0x7f110019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 -- 172.21.15.23:0/2609766050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 msgr2=0x7f110019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 0x7f110019f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 -- 172.21.15.23:0/2609766050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001002d0 msgr2=0x7f1100198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001002d0 0x7f1100198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 -- 172.21.15.23:0/2609766050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00095f0 con 0x7f11001014c0 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10feffd700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d980 tx=0x7f10f400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4004220 con 0x7f11001014c0 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10f4007640 con 0x7f11001014c0 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110010bca0 con 0x7f11001014c0 2023-11-16T23:59:20.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4005690 con 0x7f11001014c0 2023-11-16T23:59:20.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.364+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110010bb10 con 0x7f11001014c0 2023-11-16T23:59:20.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.365+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1100192a00 con 0x7f11001014c0 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.367+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f10f4004750 con 0x7f11001014c0 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.369+0000 7f10fcff9700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 0x7f10e80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.369+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f10f40cc8b0 con 0x7f11001014c0 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.369+0000 7f10ff7fe700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 0x7f10e80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.370+0000 7f10ff7fe700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 0x7f10e80a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10f000b2e0 tx=0x7f10f0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:20.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.373+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f10f4095db0 con 0x7f11001014c0 2023-11-16T23:59:20.583 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.581+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1100061c40 con 0x7f11001014c0 2023-11-16T23:59:20.584 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.581+0000 7f10fcff9700 1 -- 172.21.15.23:0/2609766050 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f10f4016020 con 0x7f11001014c0 2023-11-16T23:59:20.585 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-16T23:59:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.583+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 msgr2=0x7f10e80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.583+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 0x7f10e80a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f10f000b2e0 tx=0x7f10f0005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 msgr2=0x7f1100198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d980 tx=0x7f10f400dc90 comp rx=0 tx=0).stop 2023-11-16T23:59:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 shutdown_connections 2023-11-16T23:59:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f10e80a11d0 0x7f10e80a3680 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1100101e50 0x7f110019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11001002d0 0x7f1100198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 --2- 172.21.15.23:0/2609766050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11001014c0 0x7f1100198890 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 >> 172.21.15.23:0/2609766050 conn(0x7f11000fb8b0 msgr2=0x7f1100106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 shutdown_connections 2023-11-16T23:59:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.584+0000 7f1105bff700 1 -- 172.21.15.23:0/2609766050 wait complete. 2023-11-16T23:59:20.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 -- 172.21.15.23:0/2919230633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 msgr2=0x7f673c108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 --2- 172.21.15.23:0/2919230633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c108670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f673400b0a0 tx=0x7f673400b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 -- 172.21.15.23:0/2919230633 shutdown_connections 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 --2- 172.21.15.23:0/2919230633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c108670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 --2- 172.21.15.23:0/2919230633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c100230 0x7f673c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 --2- 172.21.15.23:0/2919230633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c101420 0x7f673c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 -- 172.21.15.23:0/2919230633 >> 172.21.15.23:0/2919230633 conn(0x7f673c0fb810 msgr2=0x7f673c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:20.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 -- 172.21.15.23:0/2919230633 shutdown_connections 2023-11-16T23:59:20.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.817+0000 7f67443ff700 1 -- 172.21.15.23:0/2919230633 wait complete. 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.818+0000 7f67443ff700 1 Processor -- start 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.818+0000 7f67443ff700 1 -- start start 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 0x7f673c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c073bc0 con 0x7f673c101db0 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c073d30 con 0x7f673c101420 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f67443ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c1a0100 con 0x7f673c100230 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674299c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:34720/0 (socket says 172.21.15.23:34720) 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 -- 172.21.15.23:0/1546915245 learned_addr learned my addr 172.21.15.23:0/1546915245 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674199a700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 0x7f673c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 -- 172.21.15.23:0/1546915245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 msgr2=0x7f673c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 0x7f673c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 -- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 msgr2=0x7f673c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674219b700 1 -- 172.21.15.23:0/1546915245 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009d00 con 0x7f673c100230 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674199a700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 0x7f673c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.819+0000 7f674299c700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f674219b700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f673800e970 tx=0x7f673800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738009790 con 0x7f673c100230 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f673800eec0 con 0x7f673c100230 2023-11-16T23:59:20.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738003cf0 con 0x7f673c100230 2023-11-16T23:59:20.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c10bc00 con 0x7f673c100230 2023-11-16T23:59:20.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.820+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c10ba70 con 0x7f673c100230 2023-11-16T23:59:20.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.821+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c1a0820 con 0x7f673c100230 2023-11-16T23:59:20.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.823+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f673800ccd0 con 0x7f673c100230 2023-11-16T23:59:20.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.824+0000 7f67337fe700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 0x7f67280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:20.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.824+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6738014070 con 0x7f673c100230 2023-11-16T23:59:20.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.824+0000 7f674199a700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 0x7f67280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:20.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.825+0000 7f674199a700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 0x7f67280a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f672c009cb0 tx=0x7f672c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:20.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:20.829+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6738096210 con 0x7f673c100230 2023-11-16T23:59:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:20 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1127472151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:20 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2609766050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.056+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f673c107090 con 0x7f673c100230 2023-11-16T23:59:21.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.056+0000 7f67337fe700 1 -- 172.21.15.23:0/1546915245 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+591 (secure 0 0 0) 0x7f6738099830 con 0x7f673c100230 2023-11-16T23:59:21.058 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2730190791 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:21.058 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:21.058 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:21.058 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4236499031 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2789478528 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3874168431 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/792254178 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:21.059 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1353533979 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:21.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 msgr2=0x7f67280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 0x7f67280a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f672c009cb0 tx=0x7f672c009260 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 msgr2=0x7f673c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f673800e970 tx=0x7f673800ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 shutdown_connections 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f673c101db0 0x7f673c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f67280a11d0 0x7f67280a3680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c100230 0x7f673c0748a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 --2- 172.21.15.23:0/1546915245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f673c101420 0x7f673c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 >> 172.21.15.23:0/1546915245 conn(0x7f673c0fb810 msgr2=0x7f673c0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:21.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 shutdown_connections 2023-11-16T23:59:21.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.059+0000 7f67443ff700 1 -- 172.21.15.23:0/1546915245 wait complete. 2023-11-16T23:59:21.062 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2023-11-16T23:59:21.079 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:59:21.079 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T23:59:21.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:20 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1127472151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:20 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2609766050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:20 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1127472151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:20 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2609766050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:21.124 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:59:21.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 -- 172.21.15.23:0/492266465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564100290 msgr2=0x7f25641006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 --2- 172.21.15.23:0/492266465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564100290 0x7f25641006a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2554009990 tx=0x7f2554009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 -- 172.21.15.23:0/492266465 shutdown_connections 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 --2- 172.21.15.23:0/492266465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564101e10 0x7f25641086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 --2- 172.21.15.23:0/492266465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564100290 0x7f25641006a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 --2- 172.21.15.23:0/492266465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 -- 172.21.15.23:0/492266465 >> 172.21.15.23:0/492266465 conn(0x7f25640fb870 msgr2=0x7f25640fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.366+0000 7f25699ff700 1 -- 172.21.15.23:0/492266465 shutdown_connections 2023-11-16T23:59:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.367+0000 7f25699ff700 1 -- 172.21.15.23:0/492266465 wait complete. 2023-11-16T23:59:21.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.367+0000 7f25699ff700 1 Processor -- start 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 -- start start 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 0x7f2564198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564101e10 0x7f256419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256419fce0 con 0x7f2564101480 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256419fe20 con 0x7f2564100290 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25699ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256419ffb0 con 0x7f2564101e10 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25627fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25627fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40990/0 (socket says 172.21.15.23:40990) 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25627fc700 1 -- 172.21.15.23:0/2821064530 learned_addr learned my addr 172.21.15.23:0/2821064530 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f2562ffd700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 0x7f2564198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.368+0000 7f25637fe700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564101e10 0x7f256419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 -- 172.21.15.23:0/2821064530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564101e10 msgr2=0x7f256419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564101e10 0x7f256419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 -- 172.21.15.23:0/2821064530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 msgr2=0x7f2564198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 0x7f2564198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 -- 172.21.15.23:0/2821064530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540095f0 con 0x7f2564101480 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f2562ffd700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 0x7f2564198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25627fc700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f255800e9a0 tx=0x7f255800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558009790 con 0x7f2564101480 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256410bbe0 con 0x7f2564101480 2023-11-16T23:59:21.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2558010430 con 0x7f2564101480 2023-11-16T23:59:21.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558018680 con 0x7f2564101480 2023-11-16T23:59:21.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.369+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256410ba50 con 0x7f2564101480 2023-11-16T23:59:21.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.370+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2564192870 con 0x7f2564101480 2023-11-16T23:59:21.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.372+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f25580105a0 con 0x7f2564101480 2023-11-16T23:59:21.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.374+0000 7f25689fd700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 0x7f254c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.374+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2558014070 con 0x7f2564101480 2023-11-16T23:59:21.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.374+0000 7f2562ffd700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 0x7f254c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.375+0000 7f2562ffd700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 0x7f254c0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f255400b2e0 tx=0x7f2554005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.376+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2558096600 con 0x7f2564101480 2023-11-16T23:59:21.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.585+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f256402cfe0 con 0x7f2564101480 2023-11-16T23:59:21.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.587+0000 7f25689fd700 1 -- 172.21.15.23:0/2821064530 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f2558099c20 con 0x7f2564101480 2023-11-16T23:59:21.589 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:21.589 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 msgr2=0x7f254c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 0x7f254c0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f255400b2e0 tx=0x7f2554005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 msgr2=0x7f25641986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f255800e9a0 tx=0x7f255800ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 shutdown_connections 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f254c0a1150 0x7f254c0a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2564101e10 0x7f256419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2564100290 0x7f2564198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 --2- 172.21.15.23:0/2821064530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2564101480 0x7f25641986a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 >> 172.21.15.23:0/2821064530 conn(0x7f25640fb870 msgr2=0x7f2564106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 shutdown_connections 2023-11-16T23:59:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.589+0000 7f25699ff700 1 -- 172.21.15.23:0/2821064530 wait complete. 2023-11-16T23:59:21.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T23:59:21.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T23:59:21.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 -- 172.21.15.23:0/2834402926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc30101e50 msgr2=0x7fdc30108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 --2- 172.21.15.23:0/2834402926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc30101e50 0x7fdc30108710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b0a0 tx=0x7fdc2800b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 -- 172.21.15.23:0/2834402926 shutdown_connections 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 --2- 172.21.15.23:0/2834402926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc30101e50 0x7fdc30108710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 --2- 172.21.15.23:0/2834402926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 --2- 172.21.15.23:0/2834402926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc301014c0 0x7fdc30101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 -- 172.21.15.23:0/2834402926 >> 172.21.15.23:0/2834402926 conn(0x7fdc300fb8b0 msgr2=0x7fdc300fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 -- 172.21.15.23:0/2834402926 shutdown_connections 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.820+0000 7fdc34dff700 1 -- 172.21.15.23:0/2834402926 wait complete. 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 Processor -- start 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 -- start start 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 0x7fdc3019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3019ff20 con 0x7fdc301014c0 2023-11-16T23:59:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc301a0060 con 0x7fdc30101e50 2023-11-16T23:59:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.821+0000 7fdc34dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc301a01f0 con 0x7fdc301002d0 2023-11-16T23:59:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:42044/0 (socket says 172.21.15.23:42044) 2023-11-16T23:59:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41002/0 (socket says 172.21.15.23:41002) 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 -- 172.21.15.23:0/3429838464 learned_addr learned my addr 172.21.15.23:0/3429838464 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2ed9d700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 0x7fdc3019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 -- 172.21.15.23:0/3429838464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 msgr2=0x7fdc3019f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 0x7fdc3019f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 -- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 msgr2=0x7fdc301988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 -- 172.21.15.23:0/3429838464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28009d00 con 0x7fdc301002d0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2dd9b700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2ed9d700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 0x7fdc3019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.822+0000 7fdc2e59c700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000e970 tx=0x7fdc2000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.823+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20009790 con 0x7fdc301002d0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.823+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3010bc80 con 0x7fdc301002d0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.823+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc2000eec0 con 0x7fdc301002d0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.823+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20003cf0 con 0x7fdc301002d0 2023-11-16T23:59:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.823+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3010baf0 con 0x7fdc301002d0 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.824+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc301a0940 con 0x7fdc301002d0 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.826+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdc2000ccd0 con 0x7fdc301002d0 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.827+0000 7fdc1f7fe700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 0x7fdc180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.827+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdc20014070 con 0x7fdc301002d0 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.827+0000 7fdc2dd9b700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 0x7fdc180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:21.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.828+0000 7fdc2dd9b700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 0x7fdc180a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdc24011490 tx=0x7fdc24009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:21.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:21.831+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdc200962b0 con 0x7fdc301002d0 2023-11-16T23:59:22.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:21 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1546915245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:22.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:21 smithi023 ceph-mon[97824]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:22.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:21 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2821064530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.040+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc301a0ef0 con 0x7fdc301002d0 2023-11-16T23:59:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.041+0000 7fdc1f7fe700 1 -- 172.21.15.23:0/3429838464 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc200998d0 con 0x7fdc301002d0 2023-11-16T23:59:22.043 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 msgr2=0x7fdc180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 0x7fdc180a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdc24011490 tx=0x7fdc24009260 comp rx=0 tx=0).stop 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 msgr2=0x7fdc301983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000e970 tx=0x7fdc2000ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 shutdown_connections 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc30101e50 0x7fdc3019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fdc180a1150 0x7fdc180a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc301002d0 0x7fdc301983a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 --2- 172.21.15.23:0/3429838464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc301014c0 0x7fdc301988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 >> 172.21.15.23:0/3429838464 conn(0x7fdc300fb8b0 msgr2=0x7fdc30106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 shutdown_connections 2023-11-16T23:59:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.043+0000 7fdc34dff700 1 -- 172.21.15.23:0/3429838464 wait complete. 2023-11-16T23:59:22.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T23:59:22.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:21 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1546915245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:22.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:21 smithi173 ceph-mon[131534]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:22.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:21 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2821064530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:22.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:21 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1546915245' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:21 smithi156 ceph-mon[131105]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:21 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2821064530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 -- 172.21.15.23:0/3762902503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa90101e50 msgr2=0x7ffa90108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 --2- 172.21.15.23:0/3762902503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa90101e50 0x7ffa90108710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800b0a0 tx=0x7ffa8800b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 -- 172.21.15.23:0/3762902503 shutdown_connections 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 --2- 172.21.15.23:0/3762902503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa90101e50 0x7ffa90108710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 --2- 172.21.15.23:0/3762902503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901002d0 0x7ffa901006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 --2- 172.21.15.23:0/3762902503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa901014c0 0x7ffa90101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 -- 172.21.15.23:0/3762902503 >> 172.21.15.23:0/3762902503 conn(0x7ffa900fb8b0 msgr2=0x7ffa900fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:22.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 -- 172.21.15.23:0/3762902503 shutdown_connections 2023-11-16T23:59:22.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.269+0000 7ffa96bff700 1 -- 172.21.15.23:0/3762902503 wait complete. 2023-11-16T23:59:22.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.270+0000 7ffa96bff700 1 Processor -- start 2023-11-16T23:59:22.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.270+0000 7ffa96bff700 1 -- start start 2023-11-16T23:59:22.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 0x7ffa90198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 0x7ffa9019f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9019ff90 con 0x7ffa901002d0 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901a00d0 con 0x7ffa90101e50 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa96bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901a0260 con 0x7ffa901014c0 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41008/0 (socket says 172.21.15.23:41008) 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 -- 172.21.15.23:0/1886724874 learned_addr learned my addr 172.21.15.23:0/1886724874 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa8ffff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 0x7ffa90198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9519c700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 0x7ffa9019f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 -- 172.21.15.23:0/1886724874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 msgr2=0x7ffa90198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 0x7ffa90198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 -- 172.21.15.23:0/1886724874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 msgr2=0x7ffa9019f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 0x7ffa9019f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9499b700 1 -- 172.21.15.23:0/1886724874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa88009d00 con 0x7ffa901002d0 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa8ffff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 0x7ffa90198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.271+0000 7ffa9519c700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 0x7ffa9019f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa9499b700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e970 tx=0x7ffa8400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84009790 con 0x7ffa901002d0 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa840044e0 con 0x7ffa901002d0 2023-11-16T23:59:22.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9010bca0 con 0x7ffa901002d0 2023-11-16T23:59:22.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84003e30 con 0x7ffa901002d0 2023-11-16T23:59:22.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.272+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9010bb10 con 0x7ffa901002d0 2023-11-16T23:59:22.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.273+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa901a09b0 con 0x7ffa901002d0 2023-11-16T23:59:22.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.275+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffa84003f90 con 0x7ffa901002d0 2023-11-16T23:59:22.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.276+0000 7ffa8dffb700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 0x7ffa800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.277+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa84014070 con 0x7ffa901002d0 2023-11-16T23:59:22.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.277+0000 7ffa8ffff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 0x7ffa800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.277+0000 7ffa8ffff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 0x7ffa800a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009c90 tx=0x7ffa7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:22.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.278+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ffa84096420 con 0x7ffa901002d0 2023-11-16T23:59:22.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.484+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa901a0f60 con 0x7ffa901002d0 2023-11-16T23:59:22.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.485+0000 7ffa8dffb700 1 -- 172.21.15.23:0/1886724874 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa84099a40 con 0x7ffa901002d0 2023-11-16T23:59:22.487 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 msgr2=0x7ffa800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 0x7ffa800a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009c90 tx=0x7ffa7c009260 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 msgr2=0x7ffa90198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e970 tx=0x7ffa8400ed30 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 shutdown_connections 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa90101e50 0x7ffa9019f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa901002d0 0x7ffa90198410 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa800a1150 0x7ffa800a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 --2- 172.21.15.23:0/1886724874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa901014c0 0x7ffa90198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 >> 172.21.15.23:0/1886724874 conn(0x7ffa900fb8b0 msgr2=0x7ffa900fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:22.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 shutdown_connections 2023-11-16T23:59:22.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.487+0000 7ffa96bff700 1 -- 172.21.15.23:0/1886724874 wait complete. 2023-11-16T23:59:22.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T23:59:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 -- 172.21.15.23:0/1839119862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad41014b0 msgr2=0x7f3ad4101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/1839119862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad41014b0 0x7f3ad4101900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8009990 tx=0x7f3ac8009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 -- 172.21.15.23:0/1839119862 shutdown_connections 2023-11-16T23:59:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/1839119862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad4101e40 0x7f3ad4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/1839119862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/1839119862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad41014b0 0x7f3ad4101900 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.719+0000 7f3ad99ff700 1 -- 172.21.15.23:0/1839119862 >> 172.21.15.23:0/1839119862 conn(0x7f3ad40fb8a0 msgr2=0x7f3ad40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.720+0000 7f3ad99ff700 1 -- 172.21.15.23:0/1839119862 shutdown_connections 2023-11-16T23:59:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.720+0000 7f3ad99ff700 1 -- 172.21.15.23:0/1839119862 wait complete. 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 Processor -- start 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 -- start start 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad41014b0 0x7f3ad4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad4101e40 0x7f3ad4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4073bc0 con 0x7f3ad41002c0 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4073d30 con 0x7f3ad4101e40 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad99ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad41a00a0 con 0x7f3ad41014b0 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad27fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad41014b0 0x7f3ad4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41036/0 (socket says 172.21.15.23:41036) 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad2ffd700 1 -- 172.21.15.23:0/616650538 learned_addr learned my addr 172.21.15.23:0/616650538 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.721+0000 7f3ad37fe700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad4101e40 0x7f3ad4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 -- 172.21.15.23:0/616650538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad41014b0 msgr2=0x7f3ad4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad41014b0 0x7f3ad4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 -- 172.21.15.23:0/616650538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad4101e40 msgr2=0x7f3ad4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad4101e40 0x7f3ad4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 -- 172.21.15.23:0/616650538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac80095f0 con 0x7f3ad41002c0 2023-11-16T23:59:22.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad2ffd700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e970 tx=0x7f3ac400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:22.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4009790 con 0x7f3ad41002c0 2023-11-16T23:59:22.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ac400c7e0 con 0x7f3ad41002c0 2023-11-16T23:59:22.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad410bc70 con 0x7f3ad41002c0 2023-11-16T23:59:22.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.722+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4003c70 con 0x7f3ad41002c0 2023-11-16T23:59:22.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.723+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad410bae0 con 0x7f3ad41002c0 2023-11-16T23:59:22.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.724+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad41a07f0 con 0x7f3ad41002c0 2023-11-16T23:59:22.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.726+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ac400cc50 con 0x7f3ad41002c0 2023-11-16T23:59:22.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.727+0000 7f3ad89fd700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 0x7f3abc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:22.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.727+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3ac4014070 con 0x7f3ad41002c0 2023-11-16T23:59:22.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.727+0000 7f3ad27fc700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 0x7f3abc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:22.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.728+0000 7f3ad27fc700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 0x7f3abc0a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b2e0 tx=0x7f3ac8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:22.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.731+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3ac4096210 con 0x7f3ad41002c0 2023-11-16T23:59:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3429838464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:22 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1886724874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:22.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.945+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ad4107170 con 0x7f3ad41002c0 2023-11-16T23:59:22.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.945+0000 7f3ad89fd700 1 -- 172.21.15.23:0/616650538 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ac4003dd0 con 0x7f3ad41002c0 2023-11-16T23:59:22.947 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-16T23:59:22.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.947+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 msgr2=0x7f3abc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.947+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 0x7f3abc0a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b2e0 tx=0x7f3ac8005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.947+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 msgr2=0x7f3ad40748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:22.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.947+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e970 tx=0x7f3ac400ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 shutdown_connections 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ad4101e40 0x7f3ad4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad41002c0 0x7f3ad40748a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3abc0a1150 0x7f3abc0a3600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 --2- 172.21.15.23:0/616650538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ad41014b0 0x7f3ad4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 >> 172.21.15.23:0/616650538 conn(0x7f3ad40fb8a0 msgr2=0x7f3ad40fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 shutdown_connections 2023-11-16T23:59:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:22.948+0000 7f3ad99ff700 1 -- 172.21.15.23:0/616650538 wait complete. 2023-11-16T23:59:22.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T23:59:23.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:22 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3429838464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:23.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:22 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1886724874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:23.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:22 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3429838464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T23:59:23.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:22 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1886724874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 -- 172.21.15.23:0/3801272599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81014b0 msgr2=0x7ff9d8101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 --2- 172.21.15.23:0/3801272599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81014b0 0x7ff9d8101900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009990 tx=0x7ff9c4009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 -- 172.21.15.23:0/3801272599 shutdown_connections 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 --2- 172.21.15.23:0/3801272599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 --2- 172.21.15.23:0/3801272599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81002c0 0x7ff9d81006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 --2- 172.21.15.23:0/3801272599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81014b0 0x7ff9d8101900 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 -- 172.21.15.23:0/3801272599 >> 172.21.15.23:0/3801272599 conn(0x7ff9d80fb8a0 msgr2=0x7ff9d80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 -- 172.21.15.23:0/3801272599 shutdown_connections 2023-11-16T23:59:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.174+0000 7ff9defff700 1 -- 172.21.15.23:0/3801272599 wait complete. 2023-11-16T23:59:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 Processor -- start 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 -- start start 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81002c0 0x7ff9d81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81014b0 0x7ff9d81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819fe30 con 0x7ff9d8101e40 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819ff70 con 0x7ff9d81002c0 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.175+0000 7ff9defff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d81a0100 con 0x7ff9d81014b0 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41052/0 (socket says 172.21.15.23:41052) 2023-11-16T23:59:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 -- 172.21.15.23:0/771734588 learned_addr learned my addr 172.21.15.23:0/771734588 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9d7fff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81014b0 0x7ff9d81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dcd9b700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81002c0 0x7ff9d81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 -- 172.21.15.23:0/771734588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81014b0 msgr2=0x7ff9d81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81014b0 0x7ff9d81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 -- 172.21.15.23:0/771734588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81002c0 msgr2=0x7ff9d81982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81002c0 0x7ff9d81982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 -- 172.21.15.23:0/771734588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc009520 con 0x7ff9d8101e40 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.176+0000 7ff9dd59c700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000ef30 tx=0x7ff9d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.177+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0018070 con 0x7ff9d8101e40 2023-11-16T23:59:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.177+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9d000f040 con 0x7ff9d8101e40 2023-11-16T23:59:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.177+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c40095f0 con 0x7ff9d8101e40 2023-11-16T23:59:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.177+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000cd00 con 0x7ff9d8101e40 2023-11-16T23:59:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.177+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d810bc30 con 0x7ff9d8101e40 2023-11-16T23:59:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.178+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d810bdc0 con 0x7ff9d8101e40 2023-11-16T23:59:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.180+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff9d0003a00 con 0x7ff9d8101e40 2023-11-16T23:59:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.181+0000 7ff9d5ffb700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 0x7ff9c80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.181+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff9d00ccf70 con 0x7ff9d8101e40 2023-11-16T23:59:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.182+0000 7ff9dcd9b700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 0x7ff9c80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.182+0000 7ff9dcd9b700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 0x7ff9c80a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8101310 tx=0x7ff9cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.183+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff9d0096500 con 0x7ff9d8101e40 2023-11-16T23:59:23.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.391+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff9d802cfe0 con 0x7ff9d8101e40 2023-11-16T23:59:23.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.391+0000 7ff9d5ffb700 1 -- 172.21.15.23:0/771734588 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff9d0099b20 con 0x7ff9d8101e40 2023-11-16T23:59:23.395 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-16T23:59:23.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 msgr2=0x7ff9c80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 0x7ff9c80a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8101310 tx=0x7ff9cc009260 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 msgr2=0x7ff9d819f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000ef30 tx=0x7ff9d000c3a0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 shutdown_connections 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9d8101e40 0x7ff9d819f860 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9d81002c0 0x7ff9d81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff9c80a1150 0x7ff9c80a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 --2- 172.21.15.23:0/771734588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9d81014b0 0x7ff9d81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.393+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 >> 172.21.15.23:0/771734588 conn(0x7ff9d80fb8a0 msgr2=0x7ff9d8106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.394+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 shutdown_connections 2023-11-16T23:59:23.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.394+0000 7ff9defff700 1 -- 172.21.15.23:0/771734588 wait complete. 2023-11-16T23:59:23.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 -- 172.21.15.23:0/491833873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51801035b0 msgr2=0x7f5180105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 --2- 172.21.15.23:0/491833873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51801035b0 0x7f5180105990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5174009990 tx=0x7f5174009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 -- 172.21.15.23:0/491833873 shutdown_connections 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 --2- 172.21.15.23:0/491833873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5180105ed0 0x7f5180108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 --2- 172.21.15.23:0/491833873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 0x7f5180103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 --2- 172.21.15.23:0/491833873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51801035b0 0x7f5180105990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 -- 172.21.15.23:0/491833873 >> 172.21.15.23:0/491833873 conn(0x7f51800fa6f0 msgr2=0x7f51800fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 -- 172.21.15.23:0/491833873 shutdown_connections 2023-11-16T23:59:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.618+0000 7f51853ff700 1 -- 172.21.15.23:0/491833873 wait complete. 2023-11-16T23:59:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.619+0000 7f51853ff700 1 Processor -- start 2023-11-16T23:59:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 -- start start 2023-11-16T23:59:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 0x7f5180195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51801035b0 0x7f51801964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019dae0 con 0x7f5180105ed0 2023-11-16T23:59:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019dc20 con 0x7f5180100c90 2023-11-16T23:59:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f51853ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019ddb0 con 0x7f51801035b0 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f517f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f517e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51801035b0 0x7f51801964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f517f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41068/0 (socket says 172.21.15.23:41068) 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f517f7fe700 1 -- 172.21.15.23:0/3127305943 learned_addr learned my addr 172.21.15.23:0/3127305943 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.620+0000 7f517effd700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 0x7f5180195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 -- 172.21.15.23:0/3127305943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51801035b0 msgr2=0x7f51801964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51801035b0 0x7f51801964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 -- 172.21.15.23:0/3127305943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 msgr2=0x7f5180195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 0x7f5180195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 -- 172.21.15.23:0/3127305943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51740095f0 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f517f7fe700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f517800ef30 tx=0x7f517800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178018070 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f517800f040 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518010bc20 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517800cd00 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.621+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518010ba90 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.622+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51800fc260 con 0x7f5180105ed0 2023-11-16T23:59:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.624+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5178003a00 con 0x7f5180105ed0 2023-11-16T23:59:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.626+0000 7f5167fff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 0x7f51680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.626+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f51780ccf60 con 0x7f5180105ed0 2023-11-16T23:59:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.626+0000 7f517effd700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 0x7f51680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.627+0000 7f517effd700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 0x7f51680a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5170009c60 tx=0x7f5170009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.628+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f51780964f0 con 0x7f5180105ed0 2023-11-16T23:59:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.853+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f518002cfe0 con 0x7f5180105ed0 2023-11-16T23:59:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.854+0000 7f5167fff700 1 -- 172.21.15.23:0/3127305943 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+591 (secure 0 0 0) 0x7f5178099b10 con 0x7f5180105ed0 2023-11-16T23:59:23.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2730190791 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:23.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:23.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4236499031 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2789478528 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3874168431 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/792254178 2023-11-17T23:51:04.390538+0000 2023-11-16T23:59:23.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1353533979 2023-11-17T23:51:30.957768+0000 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 msgr2=0x7f51680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 0x7f51680a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5170009c60 tx=0x7f5170009260 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 msgr2=0x7f518019d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f517800ef30 tx=0x7f517800c3a0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 shutdown_connections 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5180105ed0 0x7f518019d510 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5180100c90 0x7f5180195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51680a11d0 0x7f51680a3680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 --2- 172.21.15.23:0/3127305943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51801035b0 0x7f51801964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 >> 172.21.15.23:0/3127305943 conn(0x7f51800fa6f0 msgr2=0x7f51801019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 shutdown_connections 2023-11-16T23:59:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:23.856+0000 7f51853ff700 1 -- 172.21.15.23:0/3127305943 wait complete. 2023-11-16T23:59:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2023-11-16T23:59:23.876 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2023-11-16T23:59:23.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:23.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-16T23:59:23.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[134396]: fuse finished with error 0 and tester_r 0 2023-11-16T23:59:23.947 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T23:59:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:23 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/616650538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:23 smithi023 ceph-mon[97824]: pgmap v314: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:24.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:23 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/771734588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:23 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/616650538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:23 smithi173 ceph-mon[131534]: pgmap v314: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:24.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:23 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/771734588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:24.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/616650538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T23:59:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:23 smithi156 ceph-mon[131105]: pgmap v314: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-11-16T23:59:24.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:23 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/771734588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T23:59:25.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:24 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3127305943' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:25.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:24 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3127305943' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:25.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:24 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3127305943' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T23:59:26.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:25 smithi023 ceph-mon[97824]: pgmap v315: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:26.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:25 smithi173 ceph-mon[131534]: pgmap v315: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:26.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:25 smithi156 ceph-mon[131105]: pgmap v315: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:28 smithi023 ceph-mon[97824]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:28 smithi173 ceph-mon[131534]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:28 smithi156 ceph-mon[131105]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:30.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:29 smithi023 ceph-mon[97824]: pgmap v317: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:30.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2023-11-16T23:59:30.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T23:59:30.050 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-16T23:59:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:29 smithi173 ceph-mon[131534]: pgmap v317: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:29 smithi156 ceph-mon[131105]: pgmap v317: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:30.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 -- 172.21.15.23:0/3628676410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e4101e30 msgr2=0x7fd8e41086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/3628676410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e4101e30 0x7fd8e41086f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00b0a0 tx=0x7fd8dc00b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 -- 172.21.15.23:0/3628676410 shutdown_connections 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/3628676410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e4101e30 0x7fd8e41086f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/3628676410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 0x7fd8e41006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/3628676410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e41014a0 0x7fd8e41018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 -- 172.21.15.23:0/3628676410 >> 172.21.15.23:0/3628676410 conn(0x7fd8e40fb890 msgr2=0x7fd8e40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 -- 172.21.15.23:0/3628676410 shutdown_connections 2023-11-16T23:59:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.327+0000 7fd8e9fff700 1 -- 172.21.15.23:0/3628676410 wait complete. 2023-11-16T23:59:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.328+0000 7fd8e9fff700 1 Processor -- start 2023-11-16T23:59:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.328+0000 7fd8e9fff700 1 -- start start 2023-11-16T23:59:30.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 0x7fd8e40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e4101e30 0x7fd8e4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4073bc0 con 0x7fd8e41014a0 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4073d30 con 0x7fd8e4101e30 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e9fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e41a00f0 con 0x7fd8e41002b0 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41100/0 (socket says 172.21.15.23:41100) 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 0x7fd8e40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 -- 172.21.15.23:0/1878455301 learned_addr learned my addr 172.21.15.23:0/1878455301 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8e3fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e4101e30 0x7fd8e4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 -- 172.21.15.23:0/1878455301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 msgr2=0x7fd8e40748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 0x7fd8e40748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 -- 172.21.15.23:0/1878455301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e4101e30 msgr2=0x7fd8e4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e4101e30 0x7fd8e4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.329+0000 7fd8dbfff700 1 -- 172.21.15.23:0/1878455301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc009d00 con 0x7fd8e41014a0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8dbfff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e9a0 tx=0x7fd8cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc009790 con 0x7fd8e41014a0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8cc010430 con 0x7fd8e41014a0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e410bc80 con 0x7fd8e41014a0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc018680 con 0x7fd8e41014a0 2023-11-16T23:59:30.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.330+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e410baf0 con 0x7fd8e41014a0 2023-11-16T23:59:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.331+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e41077d0 con 0x7fd8e41014a0 2023-11-16T23:59:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.333+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd8cc0105a0 con 0x7fd8e41014a0 2023-11-16T23:59:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.334+0000 7fd8e17fa700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 0x7fd8d00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.335+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd8cc014070 con 0x7fd8e41014a0 2023-11-16T23:59:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.335+0000 7fd8e37fe700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 0x7fd8d00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.335+0000 7fd8e37fe700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 0x7fd8d00a3680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4101300 tx=0x7fd8d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:30.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.339+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd8cc051730 con 0x7fd8e41014a0 2023-11-16T23:59:30.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.552+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e402cfe0 con 0x7fd8e41014a0 2023-11-16T23:59:30.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.554+0000 7fd8e17fa700 1 -- 172.21.15.23:0/1878455301 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fd8cc0961d0 con 0x7fd8e41014a0 2023-11-16T23:59:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:30.557 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:30.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.556+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 msgr2=0x7fd8d00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.556+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 0x7fd8d00a3680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4101300 tx=0x7fd8d4009260 comp rx=0 tx=0).stop 2023-11-16T23:59:30.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.556+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 msgr2=0x7fd8e4072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.556+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e9a0 tx=0x7fd8cc00ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 shutdown_connections 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8e4101e30 0x7fd8e4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd8d00a11d0 0x7fd8d00a3680 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e41002b0 0x7fd8e40748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 --2- 172.21.15.23:0/1878455301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e41014a0 0x7fd8e4072ee0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 >> 172.21.15.23:0/1878455301 conn(0x7fd8e40fb890 msgr2=0x7fd8e4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 shutdown_connections 2023-11-16T23:59:30.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.557+0000 7fd8e9fff700 1 -- 172.21.15.23:0/1878455301 wait complete. 2023-11-16T23:59:30.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 -- 172.21.15.23:0/2599013356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41002b0 msgr2=0x7f59a41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/2599013356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41002b0 0x7f59a41006c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009990 tx=0x7f59a0009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 -- 172.21.15.23:0/2599013356 shutdown_connections 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/2599013356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 0x7f59a41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/2599013356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41002b0 0x7f59a41006c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/2599013356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41014a0 0x7f59a41018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 -- 172.21.15.23:0/2599013356 >> 172.21.15.23:0/2599013356 conn(0x7f59a40fb890 msgr2=0x7f59a40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 -- 172.21.15.23:0/2599013356 shutdown_connections 2023-11-16T23:59:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.787+0000 7f59ac1ff700 1 -- 172.21.15.23:0/2599013356 wait complete. 2023-11-16T23:59:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.788+0000 7f59ac1ff700 1 Processor -- start 2023-11-16T23:59:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.788+0000 7f59ac1ff700 1 -- start start 2023-11-16T23:59:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41002b0 0x7f59a4195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 0x7f59a419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a419db30 con 0x7f59a41014a0 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a419dc70 con 0x7f59a41002b0 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59ac1ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a419de00 con 0x7f59a4101e30 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59a979a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59a979a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41114/0 (socket says 172.21.15.23:41114) 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59a979a700 1 -- 172.21.15.23:0/4257941733 learned_addr learned my addr 172.21.15.23:0/4257941733 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59a9f9b700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41002b0 0x7f59a4195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.789+0000 7f59aa79c700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 0x7f59a419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 -- 172.21.15.23:0/4257941733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 msgr2=0x7f59a419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 0x7f59a419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 -- 172.21.15.23:0/4257941733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41002b0 msgr2=0x7f59a4195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41002b0 0x7f59a4195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 -- 172.21.15.23:0/4257941733 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00095f0 con 0x7f59a41014a0 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59a979a700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f599400e9a0 tx=0x7f599400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994009790 con 0x7f59a41014a0 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5994010430 con 0x7f59a41014a0 2023-11-16T23:59:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994018680 con 0x7f59a41014a0 2023-11-16T23:59:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a410bc80 con 0x7f59a41014a0 2023-11-16T23:59:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.790+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a410baf0 con 0x7f59a41014a0 2023-11-16T23:59:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.791+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a4107230 con 0x7f59a41014a0 2023-11-16T23:59:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.793+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f59940105a0 con 0x7f59a41014a0 2023-11-16T23:59:30.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.795+0000 7f599affd700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 0x7f59900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:30.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.795+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5994014070 con 0x7f59a41014a0 2023-11-16T23:59:30.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.795+0000 7f59a9f9b700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 0x7f59900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:30.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.796+0000 7f59a9f9b700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 0x7f59900a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f59a000b2e0 tx=0x7f59a0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:30.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:30.799+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5994051730 con 0x7f59a41014a0 2023-11-16T23:59:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.008+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59a4067a60 con 0x7f59a41014a0 2023-11-16T23:59:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.010+0000 7f599affd700 1 -- 172.21.15.23:0/4257941733 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f5994096220 con 0x7f59a41014a0 2023-11-16T23:59:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:31.011 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 msgr2=0x7f59900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 0x7f59900a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f59a000b2e0 tx=0x7f59a0005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 msgr2=0x7f59a41964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f599400e9a0 tx=0x7f599400ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 shutdown_connections 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f59900a1150 0x7f59900a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a4101e30 0x7f59a419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a41002b0 0x7f59a4195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 --2- 172.21.15.23:0/4257941733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59a41014a0 0x7f59a41964f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 >> 172.21.15.23:0/4257941733 conn(0x7f59a40fb890 msgr2=0x7f59a40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 shutdown_connections 2023-11-16T23:59:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.012+0000 7f59ac1ff700 1 -- 172.21.15.23:0/4257941733 wait complete. 2023-11-16T23:59:31.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:30 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1878455301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:31.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-16T23:59:31.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:30 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1878455301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:31.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:30 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1878455301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:31.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 -- 172.21.15.23:0/126403374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41002c0 msgr2=0x7f65b41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/126403374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41002c0 0x7f65b41006d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65a40098f0 tx=0x7f65a4009c00 comp rx=0 tx=0).stop 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 -- 172.21.15.23:0/126403374 shutdown_connections 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/126403374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4101e40 0x7f65b4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/126403374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41002c0 0x7f65b41006d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/126403374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b41014b0 0x7f65b4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 -- 172.21.15.23:0/126403374 >> 172.21.15.23:0/126403374 conn(0x7f65b40fb8a0 msgr2=0x7f65b40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 -- 172.21.15.23:0/126403374 shutdown_connections 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.247+0000 7f65ba5ff700 1 -- 172.21.15.23:0/126403374 wait complete. 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 Processor -- start 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 -- start start 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41014b0 0x7f65b41965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b4101e40 0x7f65b419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419dc20 con 0x7f65b41002c0 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419dd60 con 0x7f65b4101e40 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.248+0000 7f65ba5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b419def0 con 0x7f65b41014b0 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52728/0 (socket says 172.21.15.23:52728) 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 -- 172.21.15.23:0/1297460995 learned_addr learned my addr 172.21.15.23:0/1297460995 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b37fe700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41014b0 0x7f65b41965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b8b9c700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b4101e40 0x7f65b419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 -- 172.21.15.23:0/1297460995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41014b0 msgr2=0x7f65b41965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41014b0 0x7f65b41965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 -- 172.21.15.23:0/1297460995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b4101e40 msgr2=0x7f65b419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b4101e40 0x7f65b419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 -- 172.21.15.23:0/1297460995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4009520 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.249+0000 7f65b3fff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009fd0 tx=0x7f65a400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.250+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a401c070 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.250+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a400f7d0 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.250+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b410bc10 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.250+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4003a40 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.250+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b410ba80 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.251+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b419e540 con 0x7f65b41002c0 2023-11-16T23:59:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.253+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f65a4020030 con 0x7f65b41002c0 2023-11-16T23:59:31.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.254+0000 7f65b17fa700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 0x7f659c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.254+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f65a40cdea0 con 0x7f65b41002c0 2023-11-16T23:59:31.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.254+0000 7f65b37fe700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 0x7f659c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.255+0000 7f65b37fe700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 0x7f659c0a3680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009c60 tx=0x7f65a8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:31.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.256+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f65a4097430 con 0x7f65b41002c0 2023-11-16T23:59:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.514+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f65b4107240 con 0x7f65b41002c0 2023-11-16T23:59:31.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.713+0000 7f65b17fa700 1 -- 172.21.15.23:0/1297460995 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f65a409aa50 con 0x7f65b41002c0 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 msgr2=0x7f659c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 0x7f659c0a3680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009c60 tx=0x7f65a8009260 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 msgr2=0x7f65b41960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009fd0 tx=0x7f65a400b790 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 shutdown_connections 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b4101e40 0x7f65b419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b41002c0 0x7f65b41960a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a11d0 0x7f659c0a3680 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 --2- 172.21.15.23:0/1297460995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b41014b0 0x7f65b41965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 >> 172.21.15.23:0/1297460995 conn(0x7f65b40fb8a0 msgr2=0x7f65b40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 shutdown_connections 2023-11-16T23:59:31.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.716+0000 7f65ba5ff700 1 -- 172.21.15.23:0/1297460995 wait complete. 2023-11-16T23:59:31.736 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-11-16T23:59:31.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 -- 172.21.15.23:0/3002925255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 msgr2=0x7f5194102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 --2- 172.21.15.23:0/3002925255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f5194102920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5184009990 tx=0x7f5184009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 -- 172.21.15.23:0/3002925255 shutdown_connections 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 --2- 172.21.15.23:0/3002925255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 0x7f519410a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 --2- 172.21.15.23:0/3002925255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f5194102920 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 --2- 172.21.15.23:0/3002925255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 0x7f5194103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 -- 172.21.15.23:0/3002925255 >> 172.21.15.23:0/3002925255 conn(0x7f51940fdb30 msgr2=0x7f51940fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:31.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 -- 172.21.15.23:0/3002925255 shutdown_connections 2023-11-16T23:59:31.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.943+0000 7f5198fff700 1 -- 172.21.15.23:0/3002925255 wait complete. 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.944+0000 7f5198fff700 1 Processor -- start 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.944+0000 7f5198fff700 1 -- start start 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 0x7f5194198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 0x7f519419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519419fe50 con 0x7f5194102510 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519419ff90 con 0x7f5194104090 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5198fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51941a0120 con 0x7f5194103700 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52746/0 (socket says 172.21.15.23:52746) 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 -- 172.21.15.23:0/1213460015 learned_addr learned my addr 172.21.15.23:0/1213460015 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f518bfff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 0x7f5194198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f5192d9d700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 0x7f519419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 -- 172.21.15.23:0/1213460015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 msgr2=0x7f5194198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 0x7f5194198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 -- 172.21.15.23:0/1213460015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 msgr2=0x7f519419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 0x7f519419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.945+0000 7f519259c700 1 -- 172.21.15.23:0/1213460015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51840095f0 con 0x7f5194102510 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f519259c700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51840098e0 tx=0x7f5184004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518401d070 con 0x7f5194102510 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5184005030 con 0x7f5194102510 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519410ded0 con 0x7f5194102510 2023-11-16T23:59:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184004b50 con 0x7f5194102510 2023-11-16T23:59:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.946+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519410dd40 con 0x7f5194102510 2023-11-16T23:59:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.947+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5194109ab0 con 0x7f5194102510 2023-11-16T23:59:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.949+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5184004d30 con 0x7f5194102510 2023-11-16T23:59:31.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.950+0000 7f518b7fe700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 0x7f51800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:31.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.951+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f51840cdd20 con 0x7f5194102510 2023-11-16T23:59:31.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.951+0000 7f518bfff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 0x7f51800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:31.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.951+0000 7f518bfff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 0x7f51800a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f517c009c60 tx=0x7f517c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:31.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:31.954+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f518405a730 con 0x7f5194102510 2023-11-16T23:59:32.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:31 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4257941733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:31 smithi023 ceph-mon[97824]: pgmap v318: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:32.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:31 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T23:59:32.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:31 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4257941733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:31 smithi173 ceph-mon[131534]: pgmap v318: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:32.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:31 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T23:59:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:31 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4257941733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:31 smithi156 ceph-mon[131105]: pgmap v318: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-11-16T23:59:32.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:31 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T23:59:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.208+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f5194061700 con 0x7f5194102510 2023-11-16T23:59:32.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.715+0000 7f518b7fe700 1 -- 172.21.15.23:0/1213460015 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f518409ada0 con 0x7f5194102510 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.718+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 msgr2=0x7f51800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.718+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 0x7f51800a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f517c009c60 tx=0x7f517c009260 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.718+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 msgr2=0x7f51941982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.718+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51840098e0 tx=0x7f5184004730 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 shutdown_connections 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5194104090 0x7f519419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5194102510 0x7f51941982d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f51800a11d0 0x7f51800a3680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 --2- 172.21.15.23:0/1213460015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5194103700 0x7f5194198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 >> 172.21.15.23:0/1213460015 conn(0x7f51940fdb30 msgr2=0x7f5194106bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:32.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 shutdown_connections 2023-11-16T23:59:32.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.719+0000 7f5198fff700 1 -- 172.21.15.23:0/1213460015 wait complete. 2023-11-16T23:59:32.745 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-11-16T23:59:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 -- 172.21.15.23:0/982694468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 msgr2=0x7ff604105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 --2- 172.21.15.23:0/982694468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff604105a40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009990 tx=0x7ff5f4009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 -- 172.21.15.23:0/982694468 shutdown_connections 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 --2- 172.21.15.23:0/982694468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff6041083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 --2- 172.21.15.23:0/982694468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff604103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 --2- 172.21.15.23:0/982694468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff604105a40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 -- 172.21.15.23:0/982694468 >> 172.21.15.23:0/982694468 conn(0x7ff6040fa7c0 msgr2=0x7ff6040fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 -- 172.21.15.23:0/982694468 shutdown_connections 2023-11-16T23:59:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.956+0000 7ff60bbff700 1 -- 172.21.15.23:0/982694468 wait complete. 2023-11-16T23:59:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 Processor -- start 2023-11-16T23:59:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 -- start start 2023-11-16T23:59:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff6041982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff6041987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60419fe20 con 0x7ff604103660 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60419ff60 con 0x7ff604100d40 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60bbff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6041a00f0 con 0x7ff604105f80 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.957+0000 7ff60919a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff6041987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60999b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff6041982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60919a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff6041987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52766/0 (socket says 172.21.15.23:52766) 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60999b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff6041982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:38476/0 (socket says 172.21.15.23:38476) 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:56324/0 (socket says 172.21.15.23:56324) 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60919a700 1 -- 172.21.15.23:0/621798697 learned_addr learned my addr 172.21.15.23:0/621798697 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 -- 172.21.15.23:0/621798697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 msgr2=0x7ff6041982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff6041982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 -- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 msgr2=0x7ff6041987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff6041987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 -- 172.21.15.23:0/621798697 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40095f0 con 0x7ff604105f80 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff60a19c700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00efc0 tx=0x7ff5fc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.958+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc018070 con 0x7ff604105f80 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.959+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60410bcd0 con 0x7ff604105f80 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.959+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5fc00f040 con 0x7ff604105f80 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.959+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc00cd40 con 0x7ff604105f80 2023-11-16T23:59:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.959+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60410bb40 con 0x7ff604105f80 2023-11-16T23:59:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.960+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff604107b70 con 0x7ff604105f80 2023-11-16T23:59:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.962+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5fc003a00 con 0x7ff604105f80 2023-11-16T23:59:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.963+0000 7ff5faffd700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 0x7ff5f00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.963+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff5fc0cd010 con 0x7ff604105f80 2023-11-16T23:59:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.963+0000 7ff60999b700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 0x7ff5f00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.964+0000 7ff60999b700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 0x7ff5f00a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff600009c60 tx=0x7ff600009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:32.965+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff5fc096600 con 0x7ff604105f80 2023-11-16T23:59:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:32 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T23:59:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:32 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:33.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:32 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T23:59:33.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:32 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T23:59:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:32 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:33.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:32 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T23:59:33.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:32 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1297460995' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T23:59:33.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:32 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:33.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:32 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T23:59:33.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.223+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff604061700 con 0x7ff604105f80 2023-11-16T23:59:33.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.702+0000 7ff5faffd700 1 -- 172.21.15.23:0/621798697 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff5fc099c20 con 0x7ff604105f80 2023-11-16T23:59:33.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 msgr2=0x7ff5f00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 0x7ff5f00a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff600009c60 tx=0x7ff600009260 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 msgr2=0x7ff60419f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00efc0 tx=0x7ff5fc00c3a0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 shutdown_connections 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5f00a1150 0x7ff5f00a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff604105f80 0x7ff60419f850 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff604100d40 0x7ff6041982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 --2- 172.21.15.23:0/621798697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff604103660 0x7ff6041987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 >> 172.21.15.23:0/621798697 conn(0x7ff6040fa7c0 msgr2=0x7ff6040ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 shutdown_connections 2023-11-16T23:59:33.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.705+0000 7ff60bbff700 1 -- 172.21.15.23:0/621798697 wait complete. 2023-11-16T23:59:33.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 -- 172.21.15.23:0/3179358285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6874074b30 msgr2=0x7f6874072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 --2- 172.21.15.23:0/3179358285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6874074b30 0x7f6874072f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6864009990 tx=0x7f6864009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 -- 172.21.15.23:0/3179358285 shutdown_connections 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 --2- 172.21.15.23:0/3179358285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f687410c8a0 0x7f687410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 --2- 172.21.15.23:0/3179358285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6874074b30 0x7f6874072f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 --2- 172.21.15.23:0/3179358285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 0x7f68740739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 -- 172.21.15.23:0/3179358285 >> 172.21.15.23:0/3179358285 conn(0x7f68740fb8b0 msgr2=0x7f68740fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:33.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 -- 172.21.15.23:0/3179358285 shutdown_connections 2023-11-16T23:59:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.941+0000 7f687a5ff700 1 -- 172.21.15.23:0/3179358285 wait complete. 2023-11-16T23:59:33.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.942+0000 7f687a5ff700 1 Processor -- start 2023-11-16T23:59:33.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 -- start start 2023-11-16T23:59:33.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 0x7f687419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:33.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6874074b30 0x7f687419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:33.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741a4220 con 0x7f687410c8a0 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741a4360 con 0x7f6874074b30 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f687a5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741a44f0 con 0x7f6874073550 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f6878b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f6878b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52790/0 (socket says 172.21.15.23:52790) 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f6878b9c700 1 -- 172.21.15.23:0/2140739452 learned_addr learned my addr 172.21.15.23:0/2140739452 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f68737fe700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6874074b30 0x7f687419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.943+0000 7f6873fff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 0x7f687419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 -- 172.21.15.23:0/2140739452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 msgr2=0x7f687419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 0x7f687419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 -- 172.21.15.23:0/2140739452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6874074b30 msgr2=0x7f687419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6874074b30 0x7f687419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 -- 172.21.15.23:0/2140739452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640095f0 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f6878b9c700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f686c00ef30 tx=0x7f686c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c018070 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6874112290 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f686c00f040 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c00cd00 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.944+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6874112100 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.946+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68741023f0 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.947+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f686c003a00 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.949+0000 7f68717fa700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 0x7f685c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.949+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f686c0cc680 con 0x7f687410c8a0 2023-11-16T23:59:33.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.949+0000 7f6873fff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 0x7f685c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:33.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.950+0000 7f6873fff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 0x7f685c0a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f686400b2e0 tx=0x7f6864005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:33.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:33.953+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f686c095c10 con 0x7f687410c8a0 2023-11-16T23:59:34.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/621798697' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:active 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: Cluster is now healthy 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-11-16T23:59:34.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:33 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:34.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/621798697' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:active 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T23:59:34.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: Cluster is now healthy 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:33 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1213460015' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T23:59:34.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/621798697' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] up:active 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: Cluster is now healthy 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-11-16T23:59:34.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:33 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:34.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.210+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6874061700 con 0x7f687410c8a0 2023-11-16T23:59:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.707+0000 7f68717fa700 1 -- 172.21.15.23:0/2140739452 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f686c099230 con 0x7f687410c8a0 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 msgr2=0x7f685c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 0x7f685c0a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f686400b2e0 tx=0x7f6864005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 msgr2=0x7f68741a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f686c00ef30 tx=0x7f686c00c3a0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 shutdown_connections 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f687410c8a0 0x7f68741a3c50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f685c0a1150 0x7f685c0a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6874073550 0x7f687419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 --2- 172.21.15.23:0/2140739452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6874074b30 0x7f687419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.710+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 >> 172.21.15.23:0/2140739452 conn(0x7f68740fb8b0 msgr2=0x7f68740fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.711+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 shutdown_connections 2023-11-16T23:59:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.711+0000 7f687a5ff700 1 -- 172.21.15.23:0/2140739452 wait complete. 2023-11-16T23:59:34.732 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-11-16T23:59:34.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-11-16T23:59:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 -- 172.21.15.23:0/3550022606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 msgr2=0x7f89601006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 --2- 172.21.15.23:0/3550022606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601006e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8950009990 tx=0x7f8950009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 -- 172.21.15.23:0/3550022606 shutdown_connections 2023-11-16T23:59:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 --2- 172.21.15.23:0/3550022606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8960101e50 0x7f8960108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 --2- 172.21.15.23:0/3550022606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601006e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 --2- 172.21.15.23:0/3550022606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89601014c0 0x7f8960101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 -- 172.21.15.23:0/3550022606 >> 172.21.15.23:0/3550022606 conn(0x7f89600fb8b0 msgr2=0x7f89600fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 -- 172.21.15.23:0/3550022606 shutdown_connections 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.940+0000 7f89669ff700 1 -- 172.21.15.23:0/3550022606 wait complete. 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.941+0000 7f89669ff700 1 Processor -- start 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 -- start start 2023-11-16T23:59:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8960101e50 0x7f896019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896019fe70 con 0x7f89601014c0 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896019ffb0 con 0x7f89601002d0 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f89669ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89601a0140 con 0x7f8960101e50 2023-11-16T23:59:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f895f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f895ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f8964f9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8960101e50 0x7f896019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f895f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52804/0 (socket says 172.21.15.23:52804) 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.942+0000 7f895f7fe700 1 -- 172.21.15.23:0/898293963 learned_addr learned my addr 172.21.15.23:0/898293963 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 -- 172.21.15.23:0/898293963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8960101e50 msgr2=0x7f896019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8960101e50 0x7f896019f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 -- 172.21.15.23:0/898293963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 msgr2=0x7f89601982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 -- 172.21.15.23:0/898293963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8958009c40 con 0x7f89601014c0 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895ffff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895f7fe700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f895400e9a0 tx=0x7f895400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954009790 con 0x7f89601014c0 2023-11-16T23:59:34.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f895400ce80 con 0x7f89601014c0 2023-11-16T23:59:34.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954018680 con 0x7f89601014c0 2023-11-16T23:59:34.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89500095f0 con 0x7f89601014c0 2023-11-16T23:59:34.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.943+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896010bca0 con 0x7f89601014c0 2023-11-16T23:59:34.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.944+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8960192a00 con 0x7f89601014c0 2023-11-16T23:59:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.946+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8954010430 con 0x7f89601014c0 2023-11-16T23:59:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.948+0000 7f895d7fa700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 0x7f89480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.948+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8954014070 con 0x7f89601014c0 2023-11-16T23:59:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.948+0000 7f895ffff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 0x7f89480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.948+0000 7f895ffff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 0x7f89480a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f895000b2e0 tx=0x7f8950005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:34.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:34.951+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8954051730 con 0x7f89601014c0 2023-11-16T23:59:35.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-11-16T23:59:35.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T23:59:35.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:active 2023-11-16T23:59:35.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T23:59:35.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:35.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:34 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T23:59:35.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:active 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:34 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-11-16T23:59:35.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T23:59:35.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] up:active 2023-11-16T23:59:35.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2140739452' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T23:59:35.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T23:59:35.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:34 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T23:59:35.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.192+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f8960060f30 con 0x7f89480a1150 2023-11-16T23:59:35.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.239+0000 7f895d7fa700 1 -- 172.21.15.23:0/898293963 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8960060f30 con 0x7f89480a1150 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 msgr2=0x7f89480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 0x7f89480a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f895000b2e0 tx=0x7f8950005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 msgr2=0x7f8960198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f895400e9a0 tx=0x7f895400ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 shutdown_connections 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f89480a1150 0x7f89480a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8960101e50 0x7f896019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89601002d0 0x7f89601982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 --2- 172.21.15.23:0/898293963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89601014c0 0x7f8960198830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 >> 172.21.15.23:0/898293963 conn(0x7f89600fb8b0 msgr2=0x7f8960106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:35.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 shutdown_connections 2023-11-16T23:59:35.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.241+0000 7f89669ff700 1 -- 172.21.15.23:0/898293963 wait complete. 2023-11-16T23:59:35.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 -- 172.21.15.23:0/4098467596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 msgr2=0x7f03c8105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 --2- 172.21.15.23:0/4098467596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c8105a40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009990 tx=0x7f03b8009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 -- 172.21.15.23:0/4098467596 shutdown_connections 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 --2- 172.21.15.23:0/4098467596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 0x7f03c81083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 --2- 172.21.15.23:0/4098467596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 --2- 172.21.15.23:0/4098467596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c8105a40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 -- 172.21.15.23:0/4098467596 >> 172.21.15.23:0/4098467596 conn(0x7f03c80fa7c0 msgr2=0x7f03c80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 -- 172.21.15.23:0/4098467596 shutdown_connections 2023-11-16T23:59:35.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.470+0000 7f03d07ff700 1 -- 172.21.15.23:0/4098467596 wait complete. 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.471+0000 7f03d07ff700 1 Processor -- start 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.471+0000 7f03d07ff700 1 -- start start 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c81960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c81965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 0x7f03c819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c819dc20 con 0x7f03c8103660 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c819dd60 con 0x7f03c8105f80 2023-11-16T23:59:35.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03d07ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c819def0 con 0x7f03c8100d40 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03ce59b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c81960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03cdd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c81965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03ce59b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c81960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:56390/0 (socket says 172.21.15.23:56390) 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03ce59b700 1 -- 172.21.15.23:0/3920081188 learned_addr learned my addr 172.21.15.23:0/3920081188 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03ced9c700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 0x7f03c819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.472+0000 7f03cdd9a700 1 -- 172.21.15.23:0/3920081188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 msgr2=0x7f03c81960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03cdd9a700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c81960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03cdd9a700 1 -- 172.21.15.23:0/3920081188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 msgr2=0x7f03c819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03cdd9a700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 0x7f03c819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03cdd9a700 1 -- 172.21.15.23:0/3920081188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b80095f0 con 0x7f03c8103660 2023-11-16T23:59:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03cdd9a700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c81965e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03b80098e0 tx=0x7f03b8005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b801d070 con 0x7f03c8103660 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b8021410 con 0x7f03c8103660 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8017650 con 0x7f03c8103660 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c810bca0 con 0x7f03c8103660 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.473+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c810bb10 con 0x7f03c8103660 2023-11-16T23:59:35.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.474+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c80fc330 con 0x7f03c8103660 2023-11-16T23:59:35.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.476+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f03b8021580 con 0x7f03c8103660 2023-11-16T23:59:35.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.477+0000 7f03bf7fe700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 0x7f03b40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.478+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f03b80cdca0 con 0x7f03c8103660 2023-11-16T23:59:35.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.478+0000 7f03ce59b700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 0x7f03b40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.478+0000 7f03ce59b700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 0x7f03b40a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009d20 tx=0x7f03c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:35.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.481+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f03b80972c0 con 0x7f03c8103660 2023-11-16T23:59:35.517 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:59:35.199+0000 7f3aba20a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T23:59:35.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:59:35.199+0000 7f3aba20a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T23:59:35.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:59:35.199+0000 7f3aba20a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T23:59:35.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:59:35.199+0000 7f3aba20a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T23:59:35.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-16T23:59:35.199+0000 7f3aba20a700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T23:59:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.727+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f03c8061cc0 con 0x7f03b40a11d0 2023-11-16T23:59:35.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.732+0000 7f03bf7fe700 1 -- 172.21.15.23:0/3920081188 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f03c8061cc0 con 0x7f03b40a11d0 2023-11-16T23:59:35.732 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/_nogroup/sv_0/49befec9-5c18-4659-b86e-d0deb0f0e750 2023-11-16T23:59:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 msgr2=0x7f03b40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 0x7f03b40a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009d20 tx=0x7f03c4009260 comp rx=0 tx=0).stop 2023-11-16T23:59:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 msgr2=0x7f03c81965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c81965e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03b80098e0 tx=0x7f03b8005720 comp rx=0 tx=0).stop 2023-11-16T23:59:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 shutdown_connections 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03c8105f80 0x7f03c819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f03b40a11d0 0x7f03b40a3680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03c8100d40 0x7f03c81960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 --2- 172.21.15.23:0/3920081188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03c8103660 0x7f03c81965e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 >> 172.21.15.23:0/3920081188 conn(0x7f03c80fa7c0 msgr2=0x7f03c81019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 shutdown_connections 2023-11-16T23:59:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.734+0000 7f03d07ff700 1 -- 172.21.15.23:0/3920081188 wait complete. 2023-11-16T23:59:35.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-11-16T23:59:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 -- 172.21.15.23:0/3037415398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964101c90 msgr2=0x7f1964108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 --2- 172.21.15.23:0/3037415398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964101c90 0x7f1964108530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f195c00b0a0 tx=0x7f195c00b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 -- 172.21.15.23:0/3037415398 shutdown_connections 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 --2- 172.21.15.23:0/3037415398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964101c90 0x7f1964108530 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 --2- 172.21.15.23:0/3037415398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964100050 0x7f1964100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 --2- 172.21.15.23:0/3037415398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19641012e0 0x7f1964101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.964+0000 7f1969f9d700 1 -- 172.21.15.23:0/3037415398 >> 172.21.15.23:0/3037415398 conn(0x7f19640fb650 msgr2=0x7f19640fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.965+0000 7f1969f9d700 1 -- 172.21.15.23:0/3037415398 shutdown_connections 2023-11-16T23:59:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.965+0000 7f1969f9d700 1 -- 172.21.15.23:0/3037415398 wait complete. 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 Processor -- start 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 -- start start 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 0x7f1964195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964101c90 0x7f196419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419da00 con 0x7f19641012e0 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419db40 con 0x7f1964101c90 2023-11-16T23:59:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1969f9d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419dcd0 con 0x7f1964100050 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1963fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1963fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52864/0 (socket says 172.21.15.23:52864) 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1963fff700 1 -- 172.21.15.23:0/2960695384 learned_addr learned my addr 172.21.15.23:0/2960695384 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.966+0000 7f1968f9b700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 0x7f1964195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f196979c700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964101c90 0x7f196419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 -- 172.21.15.23:0/2960695384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 msgr2=0x7f1964195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 0x7f1964195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 -- 172.21.15.23:0/2960695384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964101c90 msgr2=0x7f196419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964101c90 0x7f196419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 -- 172.21.15.23:0/2960695384 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c009d00 con 0x7f19641012e0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1968f9b700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 0x7f1964195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1963fff700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9a0 tx=0x7f195000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950009790 con 0x7f19641012e0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1950010430 con 0x7f19641012e0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.967+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950018680 con 0x7f19641012e0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.968+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196410ba10 con 0x7f19641012e0 2023-11-16T23:59:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.968+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196410b880 con 0x7f19641012e0 2023-11-16T23:59:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.969+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964067980 con 0x7f19641012e0 2023-11-16T23:59:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.971+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f19500105a0 con 0x7f19641012e0 2023-11-16T23:59:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.973+0000 7f1961ffb700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 0x7f19540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.973+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1950014070 con 0x7f19641012e0 2023-11-16T23:59:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.973+0000 7f1968f9b700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 0x7f19540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.974+0000 7f1968f9b700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 0x7f19540a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1958011490 tx=0x7f1958009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:35.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:35.976+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1950051730 con 0x7f19641012e0 2023-11-16T23:59:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-mon[97824]: pgmap v320: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.3 KiB/s wr, 4 op/s 2023-11-16T23:59:36.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:35 smithi023 ceph-mon[97824]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:36.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:35 smithi173 ceph-mon[131534]: pgmap v320: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.3 KiB/s wr, 4 op/s 2023-11-16T23:59:36.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:35 smithi173 ceph-mon[131534]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:36.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:35 smithi156 ceph-mon[131105]: pgmap v320: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.3 KiB/s wr, 4 op/s 2023-11-16T23:59:36.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:35 smithi156 ceph-mon[131105]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:36.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.221+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f1964060f30 con 0x7f19540a1150 2023-11-16T23:59:36.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.248+0000 7f1961ffb700 1 -- 172.21.15.23:0/2960695384 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1964060f30 con 0x7f19540a1150 2023-11-16T23:59:36.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 msgr2=0x7f19540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 0x7f19540a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1958011490 tx=0x7f1958009260 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 msgr2=0x7f19641963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9a0 tx=0x7f195000ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 shutdown_connections 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1964101c90 0x7f196419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f19540a1150 0x7f19540a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1964100050 0x7f1964195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 --2- 172.21.15.23:0/2960695384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19641012e0 0x7f19641963c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 >> 172.21.15.23:0/2960695384 conn(0x7f19640fb650 msgr2=0x7f19640fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 shutdown_connections 2023-11-16T23:59:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.250+0000 7f1969f9d700 1 -- 172.21.15.23:0/2960695384 wait complete. 2023-11-16T23:59:36.270 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3344646482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec0101e50 msgr2=0x7f5ec0108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3344646482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec0101e50 0x7f5ec0108710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800b0a0 tx=0x7f5eb800b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3344646482 shutdown_connections 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3344646482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec0101e50 0x7f5ec0108710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3344646482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 0x7f5ec01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3344646482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec01014c0 0x7f5ec0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3344646482 >> 172.21.15.23:0/3344646482 conn(0x7f5ec00fb8b0 msgr2=0x7f5ec00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3344646482 shutdown_connections 2023-11-16T23:59:36.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.477+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3344646482 wait complete. 2023-11-16T23:59:36.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 Processor -- start 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 -- start start 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 0x7f5ec00748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec0101e50 0x7f5ec0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0073bc0 con 0x7f5ec01014c0 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0073d30 con 0x7f5ec01002d0 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.478+0000 7f5ec5dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01a01c0 con 0x7f5ec0101e50 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52888/0 (socket says 172.21.15.23:52888) 2023-11-16T23:59:36.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 -- 172.21.15.23:0/3516876230 learned_addr learned my addr 172.21.15.23:0/3516876230 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5ebf7fe700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 0x7f5ec00748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5ebffff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec0101e50 0x7f5ec0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 -- 172.21.15.23:0/3516876230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec0101e50 msgr2=0x7f5ec0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec0101e50 0x7f5ec0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 -- 172.21.15.23:0/3516876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 msgr2=0x7f5ec00748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 0x7f5ec00748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 -- 172.21.15.23:0/3516876230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8009d00 con 0x7f5ec01014c0 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.479+0000 7f5eb7fff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800e9a0 tx=0x7f5ea800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.480+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8009790 con 0x7f5ec01014c0 2023-11-16T23:59:36.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.480+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ea8010430 con 0x7f5ec01014c0 2023-11-16T23:59:36.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.480+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec010bca0 con 0x7f5ec01014c0 2023-11-16T23:59:36.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.480+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8018680 con 0x7f5ec01014c0 2023-11-16T23:59:36.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.480+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec010bb10 con 0x7f5ec01014c0 2023-11-16T23:59:36.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.481+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec01077c0 con 0x7f5ec01014c0 2023-11-16T23:59:36.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.483+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5ea80105a0 con 0x7f5ec01014c0 2023-11-16T23:59:36.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.485+0000 7f5ebd7fa700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 0x7f5eac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.485+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5ea8014070 con 0x7f5ec01014c0 2023-11-16T23:59:36.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.485+0000 7f5ebf7fe700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 0x7f5eac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.486+0000 7f5ebf7fe700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 0x7f5eac0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0101320 tx=0x7f5eb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:36.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.489+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5ea8051730 con 0x7f5ec01014c0 2023-11-16T23:59:36.607 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:59:36 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: 2023-11-16T23:59:36.139+0000 7fac3172e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-16T23:59:36.607 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 16 23:59:36 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: 2023-11-16T23:59:36.139+0000 7fac3172e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-16T23:59:36.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.736+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ec0061c60 con 0x7f5eac0a1150 2023-11-16T23:59:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.740+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/3516876230 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f5ec0061c60 con 0x7f5eac0a1150 2023-11-16T23:59:36.740 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/_nogroup/sv_1/406dd57e-732c-4c7f-a2d6-e1aab9b3632b 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 msgr2=0x7f5eac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 0x7f5eac0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0101320 tx=0x7f5eb0009260 comp rx=0 tx=0).stop 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 msgr2=0x7f5ec0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800e9a0 tx=0x7f5ea800ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 shutdown_connections 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5eac0a1150 0x7f5eac0a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ec0101e50 0x7f5ec0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ec01002d0 0x7f5ec00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 --2- 172.21.15.23:0/3516876230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec01014c0 0x7f5ec0072ee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.742+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 >> 172.21.15.23:0/3516876230 conn(0x7f5ec00fb8b0 msgr2=0x7f5ec0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.743+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 shutdown_connections 2023-11-16T23:59:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.743+0000 7f5ec5dff700 1 -- 172.21.15.23:0/3516876230 wait complete. 2023-11-16T23:59:36.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T23:59:36.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 -- 172.21.15.23:0/970867359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70074ab0 msgr2=0x7f2b70072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 --2- 172.21.15.23:0/970867359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70074ab0 0x7f2b70072f00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009990 tx=0x7f2b64009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 -- 172.21.15.23:0/970867359 shutdown_connections 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 --2- 172.21.15.23:0/970867359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b70108180 0x7f2b7010a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 --2- 172.21.15.23:0/970867359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70074ab0 0x7f2b70072f00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 --2- 172.21.15.23:0/970867359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b700734d0 0x7f2b70073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 -- 172.21.15.23:0/970867359 >> 172.21.15.23:0/970867359 conn(0x7f2b700fb8b0 msgr2=0x7f2b700fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 -- 172.21.15.23:0/970867359 shutdown_connections 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.965+0000 7f2b771ff700 1 -- 172.21.15.23:0/970867359 wait complete. 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 Processor -- start 2023-11-16T23:59:36.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 -- start start 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b70074ab0 0x7f2b701986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 0x7f2b7019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.966+0000 7f2b771ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019fd10 con 0x7f2b70108180 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b771ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019fe50 con 0x7f2b70074ab0 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b771ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019ffe0 con 0x7f2b700734d0 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b7579c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 0x7f2b7019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:56438/0 (socket says 172.21.15.23:56438) 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 -- 172.21.15.23:0/36633988 learned_addr learned my addr 172.21.15.23:0/36633988 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b6ffff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b70074ab0 0x7f2b701986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 -- 172.21.15.23:0/36633988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b70074ab0 msgr2=0x7f2b701986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b70074ab0 0x7f2b701986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 -- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 msgr2=0x7f2b7019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 0x7f2b7019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 -- 172.21.15.23:0/36633988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640095f0 con 0x7f2b700734d0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b7579c700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 0x7f2b7019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.967+0000 7f2b74f9b700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b64004730 tx=0x7f2b64004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.968+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6401d070 con 0x7f2b700734d0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.968+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7010de60 con 0x7f2b700734d0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.968+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b64004c90 con 0x7f2b700734d0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.968+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64005570 con 0x7f2b700734d0 2023-11-16T23:59:36.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.968+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7010dcd0 con 0x7f2b700734d0 2023-11-16T23:59:36.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.969+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b701068c0 con 0x7f2b700734d0 2023-11-16T23:59:36.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.971+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2b6400fc70 con 0x7f2b700734d0 2023-11-16T23:59:36.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.972+0000 7f2b6dffb700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 0x7f2b600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:36.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.972+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2b640cde90 con 0x7f2b700734d0 2023-11-16T23:59:36.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.972+0000 7f2b6ffff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 0x7f2b600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:36.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.973+0000 7f2b6ffff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 0x7f2b600a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c005950 tx=0x7f2b5c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:36.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:36.974+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b64097420 con 0x7f2b700734d0 2023-11-16T23:59:37.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:36 smithi023 ceph-mon[97824]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:36 smithi023 ceph-mon[97824]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:36 smithi173 ceph-mon[131534]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:36 smithi173 ceph-mon[131534]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:36 smithi156 ceph-mon[131105]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:36 smithi156 ceph-mon[131105]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.227+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b70067a60 con 0x7f2b700734d0 2023-11-16T23:59:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.228+0000 7f2b6dffb700 1 -- 172.21.15.23:0/36633988 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7f2b6409aa40 con 0x7f2b700734d0 2023-11-16T23:59:37.230 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:37.230 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T23:59:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 msgr2=0x7f2b600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 0x7f2b600a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c005950 tx=0x7f2b5c0058e0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 msgr2=0x7f2b70198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b64004730 tx=0x7f2b64004760 comp rx=0 tx=0).stop 2023-11-16T23:59:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 shutdown_connections 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b70108180 0x7f2b7019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2b600a1150 0x7f2b600a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b700734d0 0x7f2b70198190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 --2- 172.21.15.23:0/36633988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b70074ab0 0x7f2b701986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 >> 172.21.15.23:0/36633988 conn(0x7f2b700fb8b0 msgr2=0x7f2b700fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 shutdown_connections 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.231+0000 7f2b771ff700 1 -- 172.21.15.23:0/36633988 wait complete. 2023-11-16T23:59:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-16T23:59:37.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.455+0000 7fda747ff700 1 -- 172.21.15.23:0/1314769072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c101480 msgr2=0x7fda6c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 --2- 172.21.15.23:0/1314769072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c101480 0x7fda6c1018d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009990 tx=0x7fda5c009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 -- 172.21.15.23:0/1314769072 shutdown_connections 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 --2- 172.21.15.23:0/1314769072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 0x7fda6c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 --2- 172.21.15.23:0/1314769072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c100290 0x7fda6c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 --2- 172.21.15.23:0/1314769072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c101480 0x7fda6c1018d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 -- 172.21.15.23:0/1314769072 >> 172.21.15.23:0/1314769072 conn(0x7fda6c0fb870 msgr2=0x7fda6c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 -- 172.21.15.23:0/1314769072 shutdown_connections 2023-11-16T23:59:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.456+0000 7fda747ff700 1 -- 172.21.15.23:0/1314769072 wait complete. 2023-11-16T23:59:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 Processor -- start 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 -- start start 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c101480 0x7fda6c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 0x7fda6c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c19fcb0 con 0x7fda6c100290 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c19fdf0 con 0x7fda6c101480 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda747ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c19ff80 con 0x7fda6c101e10 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda7259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda7259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52916/0 (socket says 172.21.15.23:52916) 2023-11-16T23:59:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.457+0000 7fda7259b700 1 -- 172.21.15.23:0/714881951 learned_addr learned my addr 172.21.15.23:0/714881951 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda71d9a700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c101480 0x7fda6c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda72d9c700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 0x7fda6c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 -- 172.21.15.23:0/714881951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 msgr2=0x7fda6c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 0x7fda6c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 -- 172.21.15.23:0/714881951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c101480 msgr2=0x7fda6c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c101480 0x7fda6c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 -- 172.21.15.23:0/714881951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0095f0 con 0x7fda6c100290 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda7259b700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda6800e970 tx=0x7fda6800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68009790 con 0x7fda6c100290 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda6800c7e0 con 0x7fda6c100290 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c10bc00 con 0x7fda6c100290 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.458+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68003c70 con 0x7fda6c100290 2023-11-16T23:59:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.459+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c10ba70 con 0x7fda6c100290 2023-11-16T23:59:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.460+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c192890 con 0x7fda6c100290 2023-11-16T23:59:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.462+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fda6800cc50 con 0x7fda6c100290 2023-11-16T23:59:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.463+0000 7fda637fe700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 0x7fda580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.463+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fda68014070 con 0x7fda6c100290 2023-11-16T23:59:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.463+0000 7fda71d9a700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 0x7fda580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.464+0000 7fda71d9a700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 0x7fda580a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b620 tx=0x7fda5c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.467+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fda680961f0 con 0x7fda6c100290 2023-11-16T23:59:37.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.679+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda6c02cfe0 con 0x7fda6c100290 2023-11-16T23:59:37.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.681+0000 7fda637fe700 1 -- 172.21.15.23:0/714881951 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fda68003dd0 con 0x7fda6c100290 2023-11-16T23:59:37.682 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:37.682 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 msgr2=0x7fda580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 0x7fda580a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b620 tx=0x7fda5c005960 comp rx=0 tx=0).stop 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 msgr2=0x7fda6c198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda6800e970 tx=0x7fda6800ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 shutdown_connections 2023-11-16T23:59:37.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fda580a1150 0x7fda580a3600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda6c101e10 0x7fda6c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda6c100290 0x7fda6c198130 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 --2- 172.21.15.23:0/714881951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda6c101480 0x7fda6c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 >> 172.21.15.23:0/714881951 conn(0x7fda6c0fb870 msgr2=0x7fda6c106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 shutdown_connections 2023-11-16T23:59:37.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.683+0000 7fda747ff700 1 -- 172.21.15.23:0/714881951 wait complete. 2023-11-16T23:59:37.704 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-16T23:59:37.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T23:59:37.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 -- 172.21.15.23:0/3139851059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 msgr2=0x7f47741018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 --2- 172.21.15.23:0/3139851059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f47741018f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f475c0098f0 tx=0x7f475c009c00 comp rx=0 tx=0).stop 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 -- 172.21.15.23:0/3139851059 shutdown_connections 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 --2- 172.21.15.23:0/3139851059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4774101e30 0x7f47741086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 --2- 172.21.15.23:0/3139851059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47741002b0 0x7f47741006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 --2- 172.21.15.23:0/3139851059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f47741018f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 -- 172.21.15.23:0/3139851059 >> 172.21.15.23:0/3139851059 conn(0x7f47740fb890 msgr2=0x7f47740fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 -- 172.21.15.23:0/3139851059 shutdown_connections 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.914+0000 7f4778dff700 1 -- 172.21.15.23:0/3139851059 wait complete. 2023-11-16T23:59:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.915+0000 7f4778dff700 1 Processor -- start 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.915+0000 7f4778dff700 1 -- start start 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f4774196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4774101e30 0x7f477419d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477419dba0 con 0x7f47741002b0 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477419dce0 con 0x7f47741014a0 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4778dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477419de70 con 0x7f4774101e30 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f477259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f477259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52936/0 (socket says 172.21.15.23:52936) 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f477259c700 1 -- 172.21.15.23:0/1125406117 learned_addr learned my addr 172.21.15.23:0/1125406117 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f476bfff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f4774196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.916+0000 7f4772d9d700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4774101e30 0x7f477419d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 -- 172.21.15.23:0/1125406117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4774101e30 msgr2=0x7f477419d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4774101e30 0x7f477419d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 -- 172.21.15.23:0/1125406117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 msgr2=0x7f4774196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f4774196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 -- 172.21.15.23:0/1125406117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c009d00 con 0x7f47741002b0 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f476bfff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f4774196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f477259c700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f476400e970 tx=0x7f476400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4764009790 con 0x7f47741002b0 2023-11-16T23:59:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475c009520 con 0x7f47741002b0 2023-11-16T23:59:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f476400c7e0 con 0x7f47741002b0 2023-11-16T23:59:37.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4764003c70 con 0x7f47741002b0 2023-11-16T23:59:37.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.917+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477410bc60 con 0x7f47741002b0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.918+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477410bdf0 con 0x7f47741002b0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.920+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f476400cc50 con 0x7f47741002b0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.921+0000 7f476b7fe700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 0x7f47600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.922+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4764014070 con 0x7f47741002b0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.922+0000 7f476bfff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 0x7f47600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.922+0000 7f476bfff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 0x7f47600a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f475c005e90 tx=0x7f475c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:37.926+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4764096240 con 0x7f47741002b0 2023-11-16T23:59:38.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:37 smithi023 ceph-mon[97824]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:37 smithi023 ceph-mon[97824]: pgmap v321: 129 pgs: 129 active+clean; 583 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-16T23:59:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:37 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/36633988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:37 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/714881951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:37 smithi173 ceph-mon[131534]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:38.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:37 smithi173 ceph-mon[131534]: pgmap v321: 129 pgs: 129 active+clean; 583 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-16T23:59:38.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:37 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/36633988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:37 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/714881951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:37 smithi156 ceph-mon[131105]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T23:59:38.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:37 smithi156 ceph-mon[131105]: pgmap v321: 129 pgs: 129 active+clean; 583 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-16T23:59:38.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:37 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/36633988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.112 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:37 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/714881951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.130+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4774060f30 con 0x7f47600a11e0 2023-11-16T23:59:38.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.146+0000 7f476b7fe700 1 -- 172.21.15.23:0/1125406117 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422658 (secure 0 0 0) 0x7f4774060f30 con 0x7f47600a11e0 2023-11-16T23:59:38.153 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 msgr2=0x7f47600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 0x7f47600a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f475c005e90 tx=0x7f475c019040 comp rx=0 tx=0).stop 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 msgr2=0x7f4774196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f476400e970 tx=0x7f476400ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 shutdown_connections 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f47600a11e0 0x7f47600a3690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4774101e30 0x7f477419d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47741002b0 0x7f4774196020 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 --2- 172.21.15.23:0/1125406117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47741014a0 0x7f4774196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 >> 172.21.15.23:0/1125406117 conn(0x7f47740fb890 msgr2=0x7f47740fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 shutdown_connections 2023-11-16T23:59:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.151+0000 7f4778dff700 1 -- 172.21.15.23:0/1125406117 wait complete. 2023-11-16T23:59:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-11-16T23:59:38.170 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2023-11-16T23:59:37.064960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597217,"num_objects":56,"num_object_clones":0,"num_object_copies":169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":145,"num_read_kb":128,"num_write":298,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81648,"kb_used_data":13264,"kb_used_omap":0,"kb_used_meta":68288,"kb_avail":1124860176,"statfs":{"total":1151940427776,"available":1151856820224,"internally_reserved":0,"allocated":13582336,"data_stored":7469444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69926912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2257,"num_objects":32,"num_object_clones":0,"num_object_copies":97,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":68,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003737"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241211+0000","last_change":"2023-11-16T23:58:46.280372+0000","last_active":"2023-11-16T23:58:51.241211+0000","last_peered":"2023-11-16T23:58:51.241211+0000","last_clean":"2023-11-16T23:58:51.241211+0000","last_became_active":"2023-11-16T23:58:46.280021+0000","last_became_peered":"2023-11-16T23:58:46.280021+0000","last_unstale":"2023-11-16T23:58:51.241211+0000","last_undegraded":"2023-11-16T23:58:51.241211+0000","last_fullsized":"2023-11-16T23:58:51.241211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:37.900152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243114+0000","last_change":"2023-11-16T23:58:46.297046+0000","last_active":"2023-11-16T23:58:51.243114+0000","last_peered":"2023-11-16T23:58:51.243114+0000","last_clean":"2023-11-16T23:58:51.243114+0000","last_became_active":"2023-11-16T23:58:46.282128+0000","last_became_peered":"2023-11-16T23:58:46.282128+0000","last_unstale":"2023-11-16T23:58:51.243114+0000","last_undegraded":"2023-11-16T23:58:51.243114+0000","last_fullsized":"2023-11-16T23:58:51.243114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:56.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242939+0000","last_change":"2023-11-16T23:58:46.283752+0000","last_active":"2023-11-16T23:58:51.242939+0000","last_peered":"2023-11-16T23:58:51.242939+0000","last_clean":"2023-11-16T23:58:51.242939+0000","last_became_active":"2023-11-16T23:58:46.280787+0000","last_became_peered":"2023-11-16T23:58:46.280787+0000","last_unstale":"2023-11-16T23:58:51.242939+0000","last_undegraded":"2023-11-16T23:58:51.242939+0000","last_fullsized":"2023-11-16T23:58:51.242939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:06.203820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974468+0000","last_change":"2023-11-16T23:58:46.282847+0000","last_active":"2023-11-16T23:58:51.974468+0000","last_peered":"2023-11-16T23:58:51.974468+0000","last_clean":"2023-11-16T23:58:51.974468+0000","last_became_active":"2023-11-16T23:58:46.282527+0000","last_became_peered":"2023-11-16T23:58:46.282527+0000","last_unstale":"2023-11-16T23:58:51.974468+0000","last_undegraded":"2023-11-16T23:58:51.974468+0000","last_fullsized":"2023-11-16T23:58:51.974468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:15.327419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973484+0000","last_change":"2023-11-16T23:58:46.288154+0000","last_active":"2023-11-16T23:58:51.973484+0000","last_peered":"2023-11-16T23:58:51.973484+0000","last_clean":"2023-11-16T23:58:51.973484+0000","last_became_active":"2023-11-16T23:58:46.287752+0000","last_became_peered":"2023-11-16T23:58:46.287752+0000","last_unstale":"2023-11-16T23:58:51.973484+0000","last_undegraded":"2023-11-16T23:58:51.973484+0000","last_fullsized":"2023-11-16T23:58:51.973484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:17.777171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975139+0000","last_change":"2023-11-16T23:58:46.283177+0000","last_active":"2023-11-16T23:58:51.975139+0000","last_peered":"2023-11-16T23:58:51.975139+0000","last_clean":"2023-11-16T23:58:51.975139+0000","last_became_active":"2023-11-16T23:58:46.282753+0000","last_became_peered":"2023-11-16T23:58:46.282753+0000","last_unstale":"2023-11-16T23:58:51.975139+0000","last_undegraded":"2023-11-16T23:58:51.975139+0000","last_fullsized":"2023-11-16T23:58:51.975139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:40.545830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238882+0000","last_change":"2023-11-16T23:58:46.296002+0000","last_active":"2023-11-16T23:58:51.238882+0000","last_peered":"2023-11-16T23:58:51.238882+0000","last_clean":"2023-11-16T23:58:51.238882+0000","last_became_active":"2023-11-16T23:58:46.295191+0000","last_became_peered":"2023-11-16T23:58:46.295191+0000","last_unstale":"2023-11-16T23:58:51.238882+0000","last_undegraded":"2023-11-16T23:58:51.238882+0000","last_fullsized":"2023-11-16T23:58:51.238882+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:32:43.802637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973904+0000","last_change":"2023-11-16T23:58:46.294824+0000","last_active":"2023-11-16T23:58:51.973904+0000","last_peered":"2023-11-16T23:58:51.973904+0000","last_clean":"2023-11-16T23:58:51.973904+0000","last_became_active":"2023-11-16T23:58:46.294502+0000","last_became_peered":"2023-11-16T23:58:46.294502+0000","last_unstale":"2023-11-16T23:58:51.973904+0000","last_undegraded":"2023-11-16T23:58:51.973904+0000","last_fullsized":"2023-11-16T23:58:51.973904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:47:28.853625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974608+0000","last_change":"2023-11-16T23:58:46.292583+0000","last_active":"2023-11-16T23:58:51.974608+0000","last_peered":"2023-11-16T23:58:51.974608+0000","last_clean":"2023-11-16T23:58:51.974608+0000","last_became_active":"2023-11-16T23:58:46.292089+0000","last_became_peered":"2023-11-16T23:58:46.292089+0000","last_unstale":"2023-11-16T23:58:51.974608+0000","last_undegraded":"2023-11-16T23:58:51.974608+0000","last_fullsized":"2023-11-16T23:58:51.974608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:54.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975325+0000","last_change":"2023-11-16T23:58:46.289410+0000","last_active":"2023-11-16T23:58:51.975325+0000","last_peered":"2023-11-16T23:58:51.975325+0000","last_clean":"2023-11-16T23:58:51.975325+0000","last_became_active":"2023-11-16T23:58:46.288746+0000","last_became_peered":"2023-11-16T23:58:46.288746+0000","last_unstale":"2023-11-16T23:58:51.975325+0000","last_undegraded":"2023-11-16T23:58:51.975325+0000","last_fullsized":"2023-11-16T23:58:51.975325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:39:36.326325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239625+0000","last_change":"2023-11-16T23:58:46.288862+0000","last_active":"2023-11-16T23:58:51.239625+0000","last_peered":"2023-11-16T23:58:51.239625+0000","last_clean":"2023-11-16T23:58:51.239625+0000","last_became_active":"2023-11-16T23:58:46.288046+0000","last_became_peered":"2023-11-16T23:58:46.288046+0000","last_unstale":"2023-11-16T23:58:51.239625+0000","last_undegraded":"2023-11-16T23:58:51.239625+0000","last_fullsized":"2023-11-16T23:58:51.239625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:28:13.350271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974869+0000","last_change":"2023-11-16T23:58:46.283219+0000","last_active":"2023-11-16T23:58:51.974869+0000","last_peered":"2023-11-16T23:58:51.974869+0000","last_clean":"2023-11-16T23:58:51.974869+0000","last_became_active":"2023-11-16T23:58:46.282759+0000","last_became_peered":"2023-11-16T23:58:46.282759+0000","last_unstale":"2023-11-16T23:58:51.974869+0000","last_undegraded":"2023-11-16T23:58:51.974869+0000","last_fullsized":"2023-11-16T23:58:51.974869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:02:48.010586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:46.292139+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:46.291682+0000","last_became_peered":"2023-11-16T23:58:46.291682+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:27:16.238637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242504+0000","last_change":"2023-11-16T23:58:46.293326+0000","last_active":"2023-11-16T23:58:51.242504+0000","last_peered":"2023-11-16T23:58:51.242504+0000","last_clean":"2023-11-16T23:58:51.242504+0000","last_became_active":"2023-11-16T23:58:46.286619+0000","last_became_peered":"2023-11-16T23:58:46.286619+0000","last_unstale":"2023-11-16T23:58:51.242504+0000","last_undegraded":"2023-11-16T23:58:51.242504+0000","last_fullsized":"2023-11-16T23:58:51.242504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:41.213234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242574+0000","last_change":"2023-11-16T23:58:46.274095+0000","last_active":"2023-11-16T23:58:51.242574+0000","last_peered":"2023-11-16T23:58:51.242574+0000","last_clean":"2023-11-16T23:58:51.242574+0000","last_became_active":"2023-11-16T23:58:46.272718+0000","last_became_peered":"2023-11-16T23:58:46.272718+0000","last_unstale":"2023-11-16T23:58:51.242574+0000","last_undegraded":"2023-11-16T23:58:51.242574+0000","last_fullsized":"2023-11-16T23:58:51.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:45.267014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973636+0000","last_change":"2023-11-16T23:58:46.294079+0000","last_active":"2023-11-16T23:58:51.973636+0000","last_peered":"2023-11-16T23:58:51.973636+0000","last_clean":"2023-11-16T23:58:51.973636+0000","last_became_active":"2023-11-16T23:58:46.293738+0000","last_became_peered":"2023-11-16T23:58:46.293738+0000","last_unstale":"2023-11-16T23:58:51.973636+0000","last_undegraded":"2023-11-16T23:58:51.973636+0000","last_fullsized":"2023-11-16T23:58:51.973636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:48.113990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241160+0000","last_change":"2023-11-16T23:58:46.296439+0000","last_active":"2023-11-16T23:58:51.241160+0000","last_peered":"2023-11-16T23:58:51.241160+0000","last_clean":"2023-11-16T23:58:51.241160+0000","last_became_active":"2023-11-16T23:58:46.296002+0000","last_became_peered":"2023-11-16T23:58:46.296002+0000","last_unstale":"2023-11-16T23:58:51.241160+0000","last_undegraded":"2023-11-16T23:58:51.241160+0000","last_fullsized":"2023-11-16T23:58:51.241160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:45.679213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975910+0000","last_change":"2023-11-16T23:58:46.287374+0000","last_active":"2023-11-16T23:58:51.975910+0000","last_peered":"2023-11-16T23:58:51.975910+0000","last_clean":"2023-11-16T23:58:51.975910+0000","last_became_active":"2023-11-16T23:58:46.287044+0000","last_became_peered":"2023-11-16T23:58:46.287044+0000","last_unstale":"2023-11-16T23:58:51.975910+0000","last_undegraded":"2023-11-16T23:58:51.975910+0000","last_fullsized":"2023-11-16T23:58:51.975910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:31:41.509204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416818+0000","last_change":"2023-11-16T23:58:46.284627+0000","last_active":"2023-11-16T23:58:51.416818+0000","last_peered":"2023-11-16T23:58:51.416818+0000","last_clean":"2023-11-16T23:58:51.416818+0000","last_became_active":"2023-11-16T23:58:46.284221+0000","last_became_peered":"2023-11-16T23:58:46.284221+0000","last_unstale":"2023-11-16T23:58:51.416818+0000","last_undegraded":"2023-11-16T23:58:51.416818+0000","last_fullsized":"2023-11-16T23:58:51.416818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:37.717489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416711+0000","last_change":"2023-11-16T23:58:50.742742+0000","last_active":"2023-11-16T23:58:51.416711+0000","last_peered":"2023-11-16T23:58:51.416711+0000","last_clean":"2023-11-16T23:58:51.416711+0000","last_became_active":"2023-11-16T23:58:50.742337+0000","last_became_peered":"2023-11-16T23:58:50.742337+0000","last_unstale":"2023-11-16T23:58:51.416711+0000","last_undegraded":"2023-11-16T23:58:51.416711+0000","last_fullsized":"2023-11-16T23:58:51.416711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:03:40.742862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717374+0000","last_change":"2023-11-16T23:58:45.714612+0000","last_active":"2023-11-16T23:59:33.717374+0000","last_peered":"2023-11-16T23:59:33.717374+0000","last_clean":"2023-11-16T23:59:33.717374+0000","last_became_active":"2023-11-16T23:58:45.714235+0000","last_became_peered":"2023-11-16T23:58:45.714235+0000","last_unstale":"2023-11-16T23:59:33.717374+0000","last_undegraded":"2023-11-16T23:59:33.717374+0000","last_fullsized":"2023-11-16T23:59:33.717374+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:20:19.287101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974930+0000","last_change":"2023-11-16T23:58:46.294399+0000","last_active":"2023-11-16T23:58:51.974930+0000","last_peered":"2023-11-16T23:58:51.974930+0000","last_clean":"2023-11-16T23:58:51.974930+0000","last_became_active":"2023-11-16T23:58:46.291722+0000","last_became_peered":"2023-11-16T23:58:46.291722+0000","last_unstale":"2023-11-16T23:58:51.974930+0000","last_undegraded":"2023-11-16T23:58:51.974930+0000","last_fullsized":"2023-11-16T23:58:51.974930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:58:25.110448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973733+0000","last_change":"2023-11-16T23:58:50.745764+0000","last_active":"2023-11-16T23:58:51.973733+0000","last_peered":"2023-11-16T23:58:51.973733+0000","last_clean":"2023-11-16T23:58:51.973733+0000","last_became_active":"2023-11-16T23:58:50.745411+0000","last_became_peered":"2023-11-16T23:58:50.745411+0000","last_unstale":"2023-11-16T23:58:51.973733+0000","last_undegraded":"2023-11-16T23:58:51.973733+0000","last_fullsized":"2023-11-16T23:58:51.973733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:04:04.552603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975096+0000","last_change":"2023-11-16T23:58:45.724215+0000","last_active":"2023-11-16T23:58:51.975096+0000","last_peered":"2023-11-16T23:58:51.975096+0000","last_clean":"2023-11-16T23:58:51.975096+0000","last_became_active":"2023-11-16T23:58:45.723866+0000","last_became_peered":"2023-11-16T23:58:45.723866+0000","last_unstale":"2023-11-16T23:58:51.975096+0000","last_undegraded":"2023-11-16T23:58:51.975096+0000","last_fullsized":"2023-11-16T23:58:51.975096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:09.212226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243119+0000","last_change":"2023-11-16T23:58:46.291701+0000","last_active":"2023-11-16T23:58:51.243119+0000","last_peered":"2023-11-16T23:58:51.243119+0000","last_clean":"2023-11-16T23:58:51.243119+0000","last_became_active":"2023-11-16T23:58:46.282612+0000","last_became_peered":"2023-11-16T23:58:46.282612+0000","last_unstale":"2023-11-16T23:58:51.243119+0000","last_undegraded":"2023-11-16T23:58:51.243119+0000","last_fullsized":"2023-11-16T23:58:51.243119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:18.169006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975347+0000","last_change":"2023-11-16T23:58:50.731025+0000","last_active":"2023-11-16T23:58:51.975347+0000","last_peered":"2023-11-16T23:58:51.975347+0000","last_clean":"2023-11-16T23:58:51.975347+0000","last_became_active":"2023-11-16T23:58:50.730593+0000","last_became_peered":"2023-11-16T23:58:50.730593+0000","last_unstale":"2023-11-16T23:58:51.975347+0000","last_undegraded":"2023-11-16T23:58:51.975347+0000","last_fullsized":"2023-11-16T23:58:51.975347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:18.213726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715505+0000","last_change":"2023-11-16T23:58:45.716403+0000","last_active":"2023-11-16T23:59:33.715505+0000","last_peered":"2023-11-16T23:59:33.715505+0000","last_clean":"2023-11-16T23:59:33.715505+0000","last_became_active":"2023-11-16T23:58:45.716050+0000","last_became_peered":"2023-11-16T23:58:45.716050+0000","last_unstale":"2023-11-16T23:59:33.715505+0000","last_undegraded":"2023-11-16T23:59:33.715505+0000","last_fullsized":"2023-11-16T23:59:33.715505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:42:55.310961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974544+0000","last_change":"2023-11-16T23:58:46.286460+0000","last_active":"2023-11-16T23:58:51.974544+0000","last_peered":"2023-11-16T23:58:51.974544+0000","last_clean":"2023-11-16T23:58:51.974544+0000","last_became_active":"2023-11-16T23:58:46.286135+0000","last_became_peered":"2023-11-16T23:58:46.286135+0000","last_unstale":"2023-11-16T23:58:51.974544+0000","last_undegraded":"2023-11-16T23:58:51.974544+0000","last_fullsized":"2023-11-16T23:58:51.974544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:15:02.309346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973789+0000","last_change":"2023-11-16T23:58:50.737134+0000","last_active":"2023-11-16T23:58:51.973789+0000","last_peered":"2023-11-16T23:58:51.973789+0000","last_clean":"2023-11-16T23:58:51.973789+0000","last_became_active":"2023-11-16T23:58:50.736489+0000","last_became_peered":"2023-11-16T23:58:50.736489+0000","last_unstale":"2023-11-16T23:58:51.973789+0000","last_undegraded":"2023-11-16T23:58:51.973789+0000","last_fullsized":"2023-11-16T23:58:51.973789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:33.961942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973818+0000","last_change":"2023-11-16T23:58:45.713153+0000","last_active":"2023-11-16T23:58:51.973818+0000","last_peered":"2023-11-16T23:58:51.973818+0000","last_clean":"2023-11-16T23:58:51.973818+0000","last_became_active":"2023-11-16T23:58:45.712626+0000","last_became_peered":"2023-11-16T23:58:45.712626+0000","last_unstale":"2023-11-16T23:58:51.973818+0000","last_undegraded":"2023-11-16T23:58:51.973818+0000","last_fullsized":"2023-11-16T23:58:51.973818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:55:36.907838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584145+0000","last_change":"2023-11-16T23:58:46.289729+0000","last_active":"2023-11-16T23:58:51.584145+0000","last_peered":"2023-11-16T23:58:51.584145+0000","last_clean":"2023-11-16T23:58:51.584145+0000","last_became_active":"2023-11-16T23:58:46.284791+0000","last_became_peered":"2023-11-16T23:58:46.284791+0000","last_unstale":"2023-11-16T23:58:51.584145+0000","last_undegraded":"2023-11-16T23:58:51.584145+0000","last_fullsized":"2023-11-16T23:58:51.584145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:13:27.839983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416962+0000","last_change":"2023-11-16T23:58:50.725585+0000","last_active":"2023-11-16T23:58:51.416962+0000","last_peered":"2023-11-16T23:58:51.416962+0000","last_clean":"2023-11-16T23:58:51.416962+0000","last_became_active":"2023-11-16T23:58:50.725188+0000","last_became_peered":"2023-11-16T23:58:50.725188+0000","last_unstale":"2023-11-16T23:58:51.416962+0000","last_undegraded":"2023-11-16T23:58:51.416962+0000","last_fullsized":"2023-11-16T23:58:51.416962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:54.017294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733611+0000","last_change":"2023-11-16T23:58:45.715656+0000","last_active":"2023-11-16T23:59:32.733611+0000","last_peered":"2023-11-16T23:59:32.733611+0000","last_clean":"2023-11-16T23:59:32.733611+0000","last_became_active":"2023-11-16T23:58:45.715331+0000","last_became_peered":"2023-11-16T23:58:45.715331+0000","last_unstale":"2023-11-16T23:59:32.733611+0000","last_undegraded":"2023-11-16T23:59:32.733611+0000","last_fullsized":"2023-11-16T23:59:32.733611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:38:28.590646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973481+0000","last_change":"2023-11-16T23:58:46.294442+0000","last_active":"2023-11-16T23:58:51.973481+0000","last_peered":"2023-11-16T23:58:51.973481+0000","last_clean":"2023-11-16T23:58:51.973481+0000","last_became_active":"2023-11-16T23:58:46.294112+0000","last_became_peered":"2023-11-16T23:58:46.294112+0000","last_unstale":"2023-11-16T23:58:51.973481+0000","last_undegraded":"2023-11-16T23:58:51.973481+0000","last_fullsized":"2023-11-16T23:58:51.973481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:16:37.966411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416678+0000","last_change":"2023-11-16T23:58:50.747104+0000","last_active":"2023-11-16T23:58:51.416678+0000","last_peered":"2023-11-16T23:58:51.416678+0000","last_clean":"2023-11-16T23:58:51.416678+0000","last_became_active":"2023-11-16T23:58:50.746684+0000","last_became_peered":"2023-11-16T23:58:50.746684+0000","last_unstale":"2023-11-16T23:58:51.416678+0000","last_undegraded":"2023-11-16T23:58:51.416678+0000","last_fullsized":"2023-11-16T23:58:51.416678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:25.519731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715891+0000","last_change":"2023-11-16T23:58:45.719225+0000","last_active":"2023-11-16T23:59:33.715891+0000","last_peered":"2023-11-16T23:59:33.715891+0000","last_clean":"2023-11-16T23:59:33.715891+0000","last_became_active":"2023-11-16T23:58:45.718815+0000","last_became_peered":"2023-11-16T23:58:45.718815+0000","last_unstale":"2023-11-16T23:59:33.715891+0000","last_undegraded":"2023-11-16T23:59:33.715891+0000","last_fullsized":"2023-11-16T23:59:33.715891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:09.302556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583967+0000","last_change":"2023-11-16T23:58:46.292334+0000","last_active":"2023-11-16T23:58:51.583967+0000","last_peered":"2023-11-16T23:58:51.583967+0000","last_clean":"2023-11-16T23:58:51.583967+0000","last_became_active":"2023-11-16T23:58:46.291882+0000","last_became_peered":"2023-11-16T23:58:46.291882+0000","last_unstale":"2023-11-16T23:58:51.583967+0000","last_undegraded":"2023-11-16T23:58:51.583967+0000","last_fullsized":"2023-11-16T23:58:51.583967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:17.751014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241151+0000","last_change":"2023-11-16T23:58:50.742737+0000","last_active":"2023-11-16T23:58:51.241151+0000","last_peered":"2023-11-16T23:58:51.241151+0000","last_clean":"2023-11-16T23:58:51.241151+0000","last_became_active":"2023-11-16T23:58:50.742257+0000","last_became_peered":"2023-11-16T23:58:50.742257+0000","last_unstale":"2023-11-16T23:58:51.241151+0000","last_undegraded":"2023-11-16T23:58:51.241151+0000","last_fullsized":"2023-11-16T23:58:51.241151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:47:46.061379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715391+0000","last_change":"2023-11-16T23:58:45.716537+0000","last_active":"2023-11-16T23:59:33.715391+0000","last_peered":"2023-11-16T23:59:33.715391+0000","last_clean":"2023-11-16T23:59:33.715391+0000","last_became_active":"2023-11-16T23:58:45.716018+0000","last_became_peered":"2023-11-16T23:58:45.716018+0000","last_unstale":"2023-11-16T23:59:33.715391+0000","last_undegraded":"2023-11-16T23:59:33.715391+0000","last_fullsized":"2023-11-16T23:59:33.715391+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:56:57.891968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973828+0000","last_change":"2023-11-16T23:58:46.288713+0000","last_active":"2023-11-16T23:58:51.973828+0000","last_peered":"2023-11-16T23:58:51.973828+0000","last_clean":"2023-11-16T23:58:51.973828+0000","last_became_active":"2023-11-16T23:58:46.288363+0000","last_became_peered":"2023-11-16T23:58:46.288363+0000","last_unstale":"2023-11-16T23:58:51.973828+0000","last_undegraded":"2023-11-16T23:58:51.973828+0000","last_fullsized":"2023-11-16T23:58:51.973828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:48:49.725514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243013+0000","last_change":"2023-11-16T23:58:50.728730+0000","last_active":"2023-11-16T23:58:51.243013+0000","last_peered":"2023-11-16T23:58:51.243013+0000","last_clean":"2023-11-16T23:58:51.243013+0000","last_became_active":"2023-11-16T23:58:50.728164+0000","last_became_peered":"2023-11-16T23:58:50.728164+0000","last_unstale":"2023-11-16T23:58:51.243013+0000","last_undegraded":"2023-11-16T23:58:51.243013+0000","last_fullsized":"2023-11-16T23:58:51.243013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:30:15.323131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583464+0000","last_change":"2023-11-16T23:58:45.713056+0000","last_active":"2023-11-16T23:58:51.583464+0000","last_peered":"2023-11-16T23:58:51.583464+0000","last_clean":"2023-11-16T23:58:51.583464+0000","last_became_active":"2023-11-16T23:58:45.712685+0000","last_became_peered":"2023-11-16T23:58:45.712685+0000","last_unstale":"2023-11-16T23:58:51.583464+0000","last_undegraded":"2023-11-16T23:58:51.583464+0000","last_fullsized":"2023-11-16T23:58:51.583464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:35.811313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974076+0000","last_change":"2023-11-16T23:58:46.291540+0000","last_active":"2023-11-16T23:58:51.974076+0000","last_peered":"2023-11-16T23:58:51.974076+0000","last_clean":"2023-11-16T23:58:51.974076+0000","last_became_active":"2023-11-16T23:58:46.289372+0000","last_became_peered":"2023-11-16T23:58:46.289372+0000","last_unstale":"2023-11-16T23:58:51.974076+0000","last_undegraded":"2023-11-16T23:58:51.974076+0000","last_fullsized":"2023-11-16T23:58:51.974076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:38:19.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974448+0000","last_change":"2023-11-16T23:58:50.729362+0000","last_active":"2023-11-16T23:58:51.974448+0000","last_peered":"2023-11-16T23:58:51.974448+0000","last_clean":"2023-11-16T23:58:51.974448+0000","last_became_active":"2023-11-16T23:58:50.729007+0000","last_became_peered":"2023-11-16T23:58:50.729007+0000","last_unstale":"2023-11-16T23:58:51.974448+0000","last_undegraded":"2023-11-16T23:58:51.974448+0000","last_fullsized":"2023-11-16T23:58:51.974448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:59.110409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734157+0000","last_change":"2023-11-16T23:58:45.717814+0000","last_active":"2023-11-16T23:59:32.734157+0000","last_peered":"2023-11-16T23:59:32.734157+0000","last_clean":"2023-11-16T23:59:32.734157+0000","last_became_active":"2023-11-16T23:58:45.717494+0000","last_became_peered":"2023-11-16T23:58:45.717494+0000","last_unstale":"2023-11-16T23:59:32.734157+0000","last_undegraded":"2023-11-16T23:59:32.734157+0000","last_fullsized":"2023-11-16T23:59:32.734157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:33:17.157376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974442+0000","last_change":"2023-11-16T23:58:46.285305+0000","last_active":"2023-11-16T23:58:51.974442+0000","last_peered":"2023-11-16T23:58:51.974442+0000","last_clean":"2023-11-16T23:58:51.974442+0000","last_became_active":"2023-11-16T23:58:46.284563+0000","last_became_peered":"2023-11-16T23:58:46.284563+0000","last_unstale":"2023-11-16T23:58:51.974442+0000","last_undegraded":"2023-11-16T23:58:51.974442+0000","last_fullsized":"2023-11-16T23:58:51.974442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:12.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973969+0000","last_change":"2023-11-16T23:58:50.740863+0000","last_active":"2023-11-16T23:58:51.973969+0000","last_peered":"2023-11-16T23:58:51.973969+0000","last_clean":"2023-11-16T23:58:51.973969+0000","last_became_active":"2023-11-16T23:58:50.740400+0000","last_became_peered":"2023-11-16T23:58:50.740400+0000","last_unstale":"2023-11-16T23:58:51.973969+0000","last_undegraded":"2023-11-16T23:58:51.973969+0000","last_fullsized":"2023-11-16T23:58:51.973969+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:52:44.691356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239474+0000","last_change":"2023-11-16T23:58:45.715701+0000","last_active":"2023-11-16T23:58:51.239474+0000","last_peered":"2023-11-16T23:58:51.239474+0000","last_clean":"2023-11-16T23:58:51.239474+0000","last_became_active":"2023-11-16T23:58:45.715273+0000","last_became_peered":"2023-11-16T23:58:45.715273+0000","last_unstale":"2023-11-16T23:58:51.239474+0000","last_undegraded":"2023-11-16T23:58:51.239474+0000","last_fullsized":"2023-11-16T23:58:51.239474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:08.019018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243105+0000","last_change":"2023-11-16T23:58:46.294740+0000","last_active":"2023-11-16T23:58:51.243105+0000","last_peered":"2023-11-16T23:58:51.243105+0000","last_clean":"2023-11-16T23:58:51.243105+0000","last_became_active":"2023-11-16T23:58:46.282681+0000","last_became_peered":"2023-11-16T23:58:46.282681+0000","last_unstale":"2023-11-16T23:58:51.243105+0000","last_undegraded":"2023-11-16T23:58:51.243105+0000","last_fullsized":"2023-11-16T23:58:51.243105+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:44.418038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975559+0000","last_change":"2023-11-16T23:58:50.730244+0000","last_active":"2023-11-16T23:58:51.975559+0000","last_peered":"2023-11-16T23:58:51.975559+0000","last_clean":"2023-11-16T23:58:51.975559+0000","last_became_active":"2023-11-16T23:58:50.729669+0000","last_became_peered":"2023-11-16T23:58:50.729669+0000","last_unstale":"2023-11-16T23:58:51.975559+0000","last_undegraded":"2023-11-16T23:58:51.975559+0000","last_fullsized":"2023-11-16T23:58:51.975559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:47:57.416754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584137+0000","last_change":"2023-11-16T23:58:45.722939+0000","last_active":"2023-11-16T23:58:51.584137+0000","last_peered":"2023-11-16T23:58:51.584137+0000","last_clean":"2023-11-16T23:58:51.584137+0000","last_became_active":"2023-11-16T23:58:45.722559+0000","last_became_peered":"2023-11-16T23:58:45.722559+0000","last_unstale":"2023-11-16T23:58:51.584137+0000","last_undegraded":"2023-11-16T23:58:51.584137+0000","last_fullsized":"2023-11-16T23:58:51.584137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:38.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974381+0000","last_change":"2023-11-16T23:58:46.285249+0000","last_active":"2023-11-16T23:58:51.974381+0000","last_peered":"2023-11-16T23:58:51.974381+0000","last_clean":"2023-11-16T23:58:51.974381+0000","last_became_active":"2023-11-16T23:58:46.284815+0000","last_became_peered":"2023-11-16T23:58:46.284815+0000","last_unstale":"2023-11-16T23:58:51.974381+0000","last_undegraded":"2023-11-16T23:58:51.974381+0000","last_fullsized":"2023-11-16T23:58:51.974381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:28.991224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974731+0000","last_change":"2023-11-16T23:58:50.734889+0000","last_active":"2023-11-16T23:58:51.974731+0000","last_peered":"2023-11-16T23:58:51.974731+0000","last_clean":"2023-11-16T23:58:51.974731+0000","last_became_active":"2023-11-16T23:58:50.734461+0000","last_became_peered":"2023-11-16T23:58:50.734461+0000","last_unstale":"2023-11-16T23:58:51.974731+0000","last_undegraded":"2023-11-16T23:58:51.974731+0000","last_fullsized":"2023-11-16T23:58:51.974731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:34:05.742581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'12","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.708283+0000","last_change":"2023-11-16T23:58:45.716428+0000","last_active":"2023-11-16T23:59:33.708283+0000","last_peered":"2023-11-16T23:59:33.708283+0000","last_clean":"2023-11-16T23:59:33.708283+0000","last_became_active":"2023-11-16T23:58:45.716076+0000","last_became_peered":"2023-11-16T23:58:45.716076+0000","last_unstale":"2023-11-16T23:59:33.708283+0000","last_undegraded":"2023-11-16T23:59:33.708283+0000","last_fullsized":"2023-11-16T23:59:33.708283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:14:06.840585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973844+0000","last_change":"2023-11-16T23:58:46.282889+0000","last_active":"2023-11-16T23:58:51.973844+0000","last_peered":"2023-11-16T23:58:51.973844+0000","last_clean":"2023-11-16T23:58:51.973844+0000","last_became_active":"2023-11-16T23:58:46.282450+0000","last_became_peered":"2023-11-16T23:58:46.282450+0000","last_unstale":"2023-11-16T23:58:51.973844+0000","last_undegraded":"2023-11-16T23:58:51.973844+0000","last_fullsized":"2023-11-16T23:58:51.973844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:08:09.378619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584460+0000","last_change":"2023-11-16T23:58:50.735438+0000","last_active":"2023-11-16T23:58:51.584460+0000","last_peered":"2023-11-16T23:58:51.584460+0000","last_clean":"2023-11-16T23:58:51.584460+0000","last_became_active":"2023-11-16T23:58:50.735040+0000","last_became_peered":"2023-11-16T23:58:50.735040+0000","last_unstale":"2023-11-16T23:58:51.584460+0000","last_undegraded":"2023-11-16T23:58:51.584460+0000","last_fullsized":"2023-11-16T23:58:51.584460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:20.496063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734379+0000","last_change":"2023-11-16T23:58:45.721161+0000","last_active":"2023-11-16T23:59:32.734379+0000","last_peered":"2023-11-16T23:59:32.734379+0000","last_clean":"2023-11-16T23:59:32.734379+0000","last_became_active":"2023-11-16T23:58:45.720826+0000","last_became_peered":"2023-11-16T23:58:45.720826+0000","last_unstale":"2023-11-16T23:59:32.734379+0000","last_undegraded":"2023-11-16T23:59:32.734379+0000","last_fullsized":"2023-11-16T23:59:32.734379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:33.101318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974461+0000","last_change":"2023-11-16T23:58:46.293152+0000","last_active":"2023-11-16T23:58:51.974461+0000","last_peered":"2023-11-16T23:58:51.974461+0000","last_clean":"2023-11-16T23:58:51.974461+0000","last_became_active":"2023-11-16T23:58:46.292830+0000","last_became_peered":"2023-11-16T23:58:46.292830+0000","last_unstale":"2023-11-16T23:58:51.974461+0000","last_undegraded":"2023-11-16T23:58:51.974461+0000","last_fullsized":"2023-11-16T23:58:51.974461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:06.803488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240824+0000","last_change":"2023-11-16T23:58:50.739327+0000","last_active":"2023-11-16T23:58:51.240824+0000","last_peered":"2023-11-16T23:58:51.240824+0000","last_clean":"2023-11-16T23:58:51.240824+0000","last_became_active":"2023-11-16T23:58:50.739024+0000","last_became_peered":"2023-11-16T23:58:50.739024+0000","last_unstale":"2023-11-16T23:58:51.240824+0000","last_undegraded":"2023-11-16T23:58:51.240824+0000","last_fullsized":"2023-11-16T23:58:51.240824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:58.095409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734178+0000","last_change":"2023-11-16T23:58:45.716730+0000","last_active":"2023-11-16T23:59:32.734178+0000","last_peered":"2023-11-16T23:59:32.734178+0000","last_clean":"2023-11-16T23:59:32.734178+0000","last_became_active":"2023-11-16T23:58:45.716380+0000","last_became_peered":"2023-11-16T23:58:45.716380+0000","last_unstale":"2023-11-16T23:59:32.734178+0000","last_undegraded":"2023-11-16T23:59:32.734178+0000","last_fullsized":"2023-11-16T23:59:32.734178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:06:26.975787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239189+0000","last_change":"2023-11-16T23:58:46.290867+0000","last_active":"2023-11-16T23:58:51.239189+0000","last_peered":"2023-11-16T23:58:51.239189+0000","last_clean":"2023-11-16T23:58:51.239189+0000","last_became_active":"2023-11-16T23:58:46.290464+0000","last_became_peered":"2023-11-16T23:58:46.290464+0000","last_unstale":"2023-11-16T23:58:51.239189+0000","last_undegraded":"2023-11-16T23:58:51.239189+0000","last_fullsized":"2023-11-16T23:58:51.239189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:23.908610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975901+0000","last_change":"2023-11-16T23:58:50.747045+0000","last_active":"2023-11-16T23:58:51.975901+0000","last_peered":"2023-11-16T23:58:51.975901+0000","last_clean":"2023-11-16T23:58:51.975901+0000","last_became_active":"2023-11-16T23:58:50.743955+0000","last_became_peered":"2023-11-16T23:58:50.743955+0000","last_unstale":"2023-11-16T23:58:51.975901+0000","last_undegraded":"2023-11-16T23:58:51.975901+0000","last_fullsized":"2023-11-16T23:58:51.975901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:54:18.403005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:34.720344+0000","last_change":"2023-11-16T23:58:45.714767+0000","last_active":"2023-11-16T23:59:34.720344+0000","last_peered":"2023-11-16T23:59:34.720344+0000","last_clean":"2023-11-16T23:59:34.720344+0000","last_became_active":"2023-11-16T23:58:45.713817+0000","last_became_peered":"2023-11-16T23:58:45.713817+0000","last_unstale":"2023-11-16T23:59:34.720344+0000","last_undegraded":"2023-11-16T23:59:34.720344+0000","last_fullsized":"2023-11-16T23:59:34.720344+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:44:48.694382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238988+0000","last_change":"2023-11-16T23:58:46.295593+0000","last_active":"2023-11-16T23:58:51.238988+0000","last_peered":"2023-11-16T23:58:51.238988+0000","last_clean":"2023-11-16T23:58:51.238988+0000","last_became_active":"2023-11-16T23:58:46.295253+0000","last_became_peered":"2023-11-16T23:58:46.295253+0000","last_unstale":"2023-11-16T23:58:51.238988+0000","last_undegraded":"2023-11-16T23:58:51.238988+0000","last_fullsized":"2023-11-16T23:58:51.238988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:57:11.390297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976616+0000","last_change":"2023-11-16T23:58:50.730370+0000","last_active":"2023-11-16T23:58:51.976616+0000","last_peered":"2023-11-16T23:58:51.976616+0000","last_clean":"2023-11-16T23:58:51.976616+0000","last_became_active":"2023-11-16T23:58:50.729915+0000","last_became_peered":"2023-11-16T23:58:50.729915+0000","last_unstale":"2023-11-16T23:58:51.976616+0000","last_undegraded":"2023-11-16T23:58:51.976616+0000","last_fullsized":"2023-11-16T23:58:51.976616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:40:19.878796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718645+0000","last_change":"2023-11-16T23:58:45.715792+0000","last_active":"2023-11-16T23:59:33.718645+0000","last_peered":"2023-11-16T23:59:33.718645+0000","last_clean":"2023-11-16T23:59:33.718645+0000","last_became_active":"2023-11-16T23:58:45.715454+0000","last_became_peered":"2023-11-16T23:58:45.715454+0000","last_unstale":"2023-11-16T23:59:33.718645+0000","last_undegraded":"2023-11-16T23:59:33.718645+0000","last_fullsized":"2023-11-16T23:59:33.718645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:51.440176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583583+0000","last_change":"2023-11-16T23:58:46.288249+0000","last_active":"2023-11-16T23:58:51.583583+0000","last_peered":"2023-11-16T23:58:51.583583+0000","last_clean":"2023-11-16T23:58:51.583583+0000","last_became_active":"2023-11-16T23:58:46.287916+0000","last_became_peered":"2023-11-16T23:58:46.287916+0000","last_unstale":"2023-11-16T23:58:51.583583+0000","last_undegraded":"2023-11-16T23:58:51.583583+0000","last_fullsized":"2023-11-16T23:58:51.583583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:34:19.800012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583795+0000","last_change":"2023-11-16T23:58:50.733835+0000","last_active":"2023-11-16T23:58:51.583795+0000","last_peered":"2023-11-16T23:58:51.583795+0000","last_clean":"2023-11-16T23:58:51.583795+0000","last_became_active":"2023-11-16T23:58:50.733373+0000","last_became_peered":"2023-11-16T23:58:50.733373+0000","last_unstale":"2023-11-16T23:58:51.583795+0000","last_undegraded":"2023-11-16T23:58:51.583795+0000","last_fullsized":"2023-11-16T23:58:51.583795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:27:17.246864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973964+0000","last_change":"2023-11-16T23:58:45.712055+0000","last_active":"2023-11-16T23:58:51.973964+0000","last_peered":"2023-11-16T23:58:51.973964+0000","last_clean":"2023-11-16T23:58:51.973964+0000","last_became_active":"2023-11-16T23:58:45.711618+0000","last_became_peered":"2023-11-16T23:58:45.711618+0000","last_unstale":"2023-11-16T23:58:51.973964+0000","last_undegraded":"2023-11-16T23:58:51.973964+0000","last_fullsized":"2023-11-16T23:58:51.973964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:37:41.700022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241050+0000","last_change":"2023-11-16T23:58:46.291460+0000","last_active":"2023-11-16T23:58:51.241050+0000","last_peered":"2023-11-16T23:58:51.241050+0000","last_clean":"2023-11-16T23:58:51.241050+0000","last_became_active":"2023-11-16T23:58:46.291144+0000","last_became_peered":"2023-11-16T23:58:46.291144+0000","last_unstale":"2023-11-16T23:58:51.241050+0000","last_undegraded":"2023-11-16T23:58:51.241050+0000","last_fullsized":"2023-11-16T23:58:51.241050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:24:10.888038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973559+0000","last_change":"2023-11-16T23:58:50.927320+0000","last_active":"2023-11-16T23:58:51.973559+0000","last_peered":"2023-11-16T23:58:51.973559+0000","last_clean":"2023-11-16T23:58:51.973559+0000","last_became_active":"2023-11-16T23:58:50.926767+0000","last_became_peered":"2023-11-16T23:58:50.926767+0000","last_unstale":"2023-11-16T23:58:51.973559+0000","last_undegraded":"2023-11-16T23:58:51.973559+0000","last_fullsized":"2023-11-16T23:58:51.973559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:21.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975281+0000","last_change":"2023-11-16T23:58:45.715654+0000","last_active":"2023-11-16T23:58:51.975281+0000","last_peered":"2023-11-16T23:58:51.975281+0000","last_clean":"2023-11-16T23:58:51.975281+0000","last_became_active":"2023-11-16T23:58:45.714899+0000","last_became_peered":"2023-11-16T23:58:45.714899+0000","last_unstale":"2023-11-16T23:58:51.975281+0000","last_undegraded":"2023-11-16T23:58:51.975281+0000","last_fullsized":"2023-11-16T23:58:51.975281+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:53.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974886+0000","last_change":"2023-11-16T23:58:46.291818+0000","last_active":"2023-11-16T23:58:51.974886+0000","last_peered":"2023-11-16T23:58:51.974886+0000","last_clean":"2023-11-16T23:58:51.974886+0000","last_became_active":"2023-11-16T23:58:46.291517+0000","last_became_peered":"2023-11-16T23:58:46.291517+0000","last_unstale":"2023-11-16T23:58:51.974886+0000","last_undegraded":"2023-11-16T23:58:51.974886+0000","last_fullsized":"2023-11-16T23:58:51.974886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:51:34.124036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:50.733546+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:50.733009+0000","last_became_peered":"2023-11-16T23:58:50.733009+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:37:16.925270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239181+0000","last_change":"2023-11-16T23:58:45.719181+0000","last_active":"2023-11-16T23:58:51.239181+0000","last_peered":"2023-11-16T23:58:51.239181+0000","last_clean":"2023-11-16T23:58:51.239181+0000","last_became_active":"2023-11-16T23:58:45.718826+0000","last_became_peered":"2023-11-16T23:58:45.718826+0000","last_unstale":"2023-11-16T23:58:51.239181+0000","last_undegraded":"2023-11-16T23:58:51.239181+0000","last_fullsized":"2023-11-16T23:58:51.239181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:52:25.180163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975356+0000","last_change":"2023-11-16T23:58:46.283405+0000","last_active":"2023-11-16T23:58:51.975356+0000","last_peered":"2023-11-16T23:58:51.975356+0000","last_clean":"2023-11-16T23:58:51.975356+0000","last_became_active":"2023-11-16T23:58:46.282995+0000","last_became_peered":"2023-11-16T23:58:46.282995+0000","last_unstale":"2023-11-16T23:58:51.975356+0000","last_undegraded":"2023-11-16T23:58:51.975356+0000","last_fullsized":"2023-11-16T23:58:51.975356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:53:28.645169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976551+0000","last_change":"2023-11-16T23:58:50.928220+0000","last_active":"2023-11-16T23:58:51.976551+0000","last_peered":"2023-11-16T23:58:51.976551+0000","last_clean":"2023-11-16T23:58:51.976551+0000","last_became_active":"2023-11-16T23:58:50.927006+0000","last_became_peered":"2023-11-16T23:58:50.927006+0000","last_unstale":"2023-11-16T23:58:51.976551+0000","last_undegraded":"2023-11-16T23:58:51.976551+0000","last_fullsized":"2023-11-16T23:58:51.976551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:09:21.135371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583555+0000","last_change":"2023-11-16T23:58:46.289260+0000","last_active":"2023-11-16T23:58:51.583555+0000","last_peered":"2023-11-16T23:58:51.583555+0000","last_clean":"2023-11-16T23:58:51.583555+0000","last_became_active":"2023-11-16T23:58:46.285041+0000","last_became_peered":"2023-11-16T23:58:46.285041+0000","last_unstale":"2023-11-16T23:58:51.583555+0000","last_undegraded":"2023-11-16T23:58:51.583555+0000","last_fullsized":"2023-11-16T23:58:51.583555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:16.675991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718821+0000","last_change":"2023-11-16T23:58:45.726423+0000","last_active":"2023-11-16T23:59:33.718821+0000","last_peered":"2023-11-16T23:59:33.718821+0000","last_clean":"2023-11-16T23:59:33.718821+0000","last_became_active":"2023-11-16T23:58:45.725996+0000","last_became_peered":"2023-11-16T23:58:45.725996+0000","last_unstale":"2023-11-16T23:59:33.718821+0000","last_undegraded":"2023-11-16T23:59:33.718821+0000","last_fullsized":"2023-11-16T23:59:33.718821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:38.058418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239536+0000","last_change":"2023-11-16T23:58:50.730148+0000","last_active":"2023-11-16T23:58:51.239536+0000","last_peered":"2023-11-16T23:58:51.239536+0000","last_clean":"2023-11-16T23:58:51.239536+0000","last_became_active":"2023-11-16T23:58:50.729675+0000","last_became_peered":"2023-11-16T23:58:50.729675+0000","last_unstale":"2023-11-16T23:58:51.239536+0000","last_undegraded":"2023-11-16T23:58:51.239536+0000","last_fullsized":"2023-11-16T23:58:51.239536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:42:47.288350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974143+0000","last_change":"2023-11-16T23:58:46.295437+0000","last_active":"2023-11-16T23:58:51.974143+0000","last_peered":"2023-11-16T23:58:51.974143+0000","last_clean":"2023-11-16T23:58:51.974143+0000","last_became_active":"2023-11-16T23:58:46.291775+0000","last_became_peered":"2023-11-16T23:58:46.291775+0000","last_unstale":"2023-11-16T23:58:51.974143+0000","last_undegraded":"2023-11-16T23:58:51.974143+0000","last_fullsized":"2023-11-16T23:58:51.974143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:17.517488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714142+0000","last_change":"2023-11-16T23:58:45.714497+0000","last_active":"2023-11-16T23:59:33.714142+0000","last_peered":"2023-11-16T23:59:33.714142+0000","last_clean":"2023-11-16T23:59:33.714142+0000","last_became_active":"2023-11-16T23:58:45.714167+0000","last_became_peered":"2023-11-16T23:58:45.714167+0000","last_unstale":"2023-11-16T23:59:33.714142+0000","last_undegraded":"2023-11-16T23:59:33.714142+0000","last_fullsized":"2023-11-16T23:59:33.714142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:46.839077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238802+0000","last_change":"2023-11-16T23:57:37.700941+0000","last_active":"2023-11-16T23:58:51.238802+0000","last_peered":"2023-11-16T23:58:51.238802+0000","last_clean":"2023-11-16T23:58:51.238802+0000","last_became_active":"2023-11-16T23:57:37.700379+0000","last_became_peered":"2023-11-16T23:57:37.700379+0000","last_unstale":"2023-11-16T23:58:51.238802+0000","last_undegraded":"2023-11-16T23:58:51.238802+0000","last_fullsized":"2023-11-16T23:58:51.238802+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_clean_scrub_stamp":"2023-11-16T23:54:03.597877+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:51:56.505655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584217+0000","last_change":"2023-11-16T23:58:50.927446+0000","last_active":"2023-11-16T23:58:51.584217+0000","last_peered":"2023-11-16T23:58:51.584217+0000","last_clean":"2023-11-16T23:58:51.584217+0000","last_became_active":"2023-11-16T23:58:50.927042+0000","last_became_peered":"2023-11-16T23:58:50.927042+0000","last_unstale":"2023-11-16T23:58:51.584217+0000","last_undegraded":"2023-11-16T23:58:51.584217+0000","last_fullsized":"2023-11-16T23:58:51.584217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:11:06.468189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973431+0000","last_change":"2023-11-16T23:58:46.283611+0000","last_active":"2023-11-16T23:58:51.973431+0000","last_peered":"2023-11-16T23:58:51.973431+0000","last_clean":"2023-11-16T23:58:51.973431+0000","last_became_active":"2023-11-16T23:58:46.281685+0000","last_became_peered":"2023-11-16T23:58:46.281685+0000","last_unstale":"2023-11-16T23:58:51.973431+0000","last_undegraded":"2023-11-16T23:58:51.973431+0000","last_fullsized":"2023-11-16T23:58:51.973431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:28:30.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238863+0000","last_change":"2023-11-16T23:58:45.716094+0000","last_active":"2023-11-16T23:58:51.238863+0000","last_peered":"2023-11-16T23:58:51.238863+0000","last_clean":"2023-11-16T23:58:51.238863+0000","last_became_active":"2023-11-16T23:58:45.715748+0000","last_became_peered":"2023-11-16T23:58:45.715748+0000","last_unstale":"2023-11-16T23:58:51.238863+0000","last_undegraded":"2023-11-16T23:58:51.238863+0000","last_fullsized":"2023-11-16T23:58:51.238863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:48.184073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240803+0000","last_change":"2023-11-16T23:58:50.737281+0000","last_active":"2023-11-16T23:58:51.240803+0000","last_peered":"2023-11-16T23:58:51.240803+0000","last_clean":"2023-11-16T23:58:51.240803+0000","last_became_active":"2023-11-16T23:58:50.736797+0000","last_became_peered":"2023-11-16T23:58:50.736797+0000","last_unstale":"2023-11-16T23:58:51.240803+0000","last_undegraded":"2023-11-16T23:58:51.240803+0000","last_fullsized":"2023-11-16T23:58:51.240803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:39.192922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734179+0000","last_change":"2023-11-16T23:58:45.722935+0000","last_active":"2023-11-16T23:59:32.734179+0000","last_peered":"2023-11-16T23:59:32.734179+0000","last_clean":"2023-11-16T23:59:32.734179+0000","last_became_active":"2023-11-16T23:58:45.722489+0000","last_became_peered":"2023-11-16T23:58:45.722489+0000","last_unstale":"2023-11-16T23:59:32.734179+0000","last_undegraded":"2023-11-16T23:59:32.734179+0000","last_fullsized":"2023-11-16T23:59:32.734179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:58.547593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975877+0000","last_change":"2023-11-16T23:58:46.284611+0000","last_active":"2023-11-16T23:58:51.975877+0000","last_peered":"2023-11-16T23:58:51.975877+0000","last_clean":"2023-11-16T23:58:51.975877+0000","last_became_active":"2023-11-16T23:58:46.283063+0000","last_became_peered":"2023-11-16T23:58:46.283063+0000","last_unstale":"2023-11-16T23:58:51.975877+0000","last_undegraded":"2023-11-16T23:58:51.975877+0000","last_fullsized":"2023-11-16T23:58:51.975877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:48:34.246739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973742+0000","last_change":"2023-11-16T23:58:50.927654+0000","last_active":"2023-11-16T23:58:51.973742+0000","last_peered":"2023-11-16T23:58:51.973742+0000","last_clean":"2023-11-16T23:58:51.973742+0000","last_became_active":"2023-11-16T23:58:50.926936+0000","last_became_peered":"2023-11-16T23:58:50.926936+0000","last_unstale":"2023-11-16T23:58:51.973742+0000","last_undegraded":"2023-11-16T23:58:51.973742+0000","last_fullsized":"2023-11-16T23:58:51.973742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:20.080380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717677+0000","last_change":"2023-11-16T23:58:45.717981+0000","last_active":"2023-11-16T23:59:33.717677+0000","last_peered":"2023-11-16T23:59:33.717677+0000","last_clean":"2023-11-16T23:59:33.717677+0000","last_became_active":"2023-11-16T23:58:45.717702+0000","last_became_peered":"2023-11-16T23:58:45.717702+0000","last_unstale":"2023-11-16T23:59:33.717677+0000","last_undegraded":"2023-11-16T23:59:33.717677+0000","last_fullsized":"2023-11-16T23:59:33.717677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:32:06.566478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974772+0000","last_change":"2023-11-16T23:58:46.288768+0000","last_active":"2023-11-16T23:58:51.974772+0000","last_peered":"2023-11-16T23:58:51.974772+0000","last_clean":"2023-11-16T23:58:51.974772+0000","last_became_active":"2023-11-16T23:58:46.288430+0000","last_became_peered":"2023-11-16T23:58:46.288430+0000","last_unstale":"2023-11-16T23:58:51.974772+0000","last_undegraded":"2023-11-16T23:58:51.974772+0000","last_fullsized":"2023-11-16T23:58:51.974772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:19:06.093201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416385+0000","last_change":"2023-11-16T23:58:50.724748+0000","last_active":"2023-11-16T23:58:51.416385+0000","last_peered":"2023-11-16T23:58:51.416385+0000","last_clean":"2023-11-16T23:58:51.416385+0000","last_became_active":"2023-11-16T23:58:50.724313+0000","last_became_peered":"2023-11-16T23:58:50.724313+0000","last_unstale":"2023-11-16T23:58:51.416385+0000","last_undegraded":"2023-11-16T23:58:51.416385+0000","last_fullsized":"2023-11-16T23:58:51.416385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:17:21.820364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732623+0000","last_change":"2023-11-16T23:58:45.715563+0000","last_active":"2023-11-16T23:59:32.732623+0000","last_peered":"2023-11-16T23:59:32.732623+0000","last_clean":"2023-11-16T23:59:32.732623+0000","last_became_active":"2023-11-16T23:58:45.715217+0000","last_became_peered":"2023-11-16T23:58:45.715217+0000","last_unstale":"2023-11-16T23:59:32.732623+0000","last_undegraded":"2023-11-16T23:59:32.732623+0000","last_fullsized":"2023-11-16T23:59:32.732623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:41.330842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242993+0000","last_change":"2023-11-16T23:58:46.298305+0000","last_active":"2023-11-16T23:58:51.242993+0000","last_peered":"2023-11-16T23:58:51.242993+0000","last_clean":"2023-11-16T23:58:51.242993+0000","last_became_active":"2023-11-16T23:58:46.297444+0000","last_became_peered":"2023-11-16T23:58:46.297444+0000","last_unstale":"2023-11-16T23:58:51.242993+0000","last_undegraded":"2023-11-16T23:58:51.242993+0000","last_fullsized":"2023-11-16T23:58:51.242993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:36.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974077+0000","last_change":"2023-11-16T23:58:50.927915+0000","last_active":"2023-11-16T23:58:51.974077+0000","last_peered":"2023-11-16T23:58:51.974077+0000","last_clean":"2023-11-16T23:58:51.974077+0000","last_became_active":"2023-11-16T23:58:50.927451+0000","last_became_peered":"2023-11-16T23:58:50.927451+0000","last_unstale":"2023-11-16T23:58:51.974077+0000","last_undegraded":"2023-11-16T23:58:51.974077+0000","last_fullsized":"2023-11-16T23:58:51.974077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:21:21.865022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732946+0000","last_change":"2023-11-16T23:58:45.710870+0000","last_active":"2023-11-16T23:59:32.732946+0000","last_peered":"2023-11-16T23:59:32.732946+0000","last_clean":"2023-11-16T23:59:32.732946+0000","last_became_active":"2023-11-16T23:58:45.710536+0000","last_became_peered":"2023-11-16T23:58:45.710536+0000","last_unstale":"2023-11-16T23:59:32.732946+0000","last_undegraded":"2023-11-16T23:59:32.732946+0000","last_fullsized":"2023-11-16T23:59:32.732946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:32.891477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973290+0000","last_change":"2023-11-16T23:58:46.282277+0000","last_active":"2023-11-16T23:58:51.973290+0000","last_peered":"2023-11-16T23:58:51.973290+0000","last_clean":"2023-11-16T23:58:51.973290+0000","last_became_active":"2023-11-16T23:58:46.281913+0000","last_became_peered":"2023-11-16T23:58:46.281913+0000","last_unstale":"2023-11-16T23:58:51.973290+0000","last_undegraded":"2023-11-16T23:58:51.973290+0000","last_fullsized":"2023-11-16T23:58:51.973290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:19:12.760160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976155+0000","last_change":"2023-11-16T23:58:50.740914+0000","last_active":"2023-11-16T23:58:51.976155+0000","last_peered":"2023-11-16T23:58:51.976155+0000","last_clean":"2023-11-16T23:58:51.976155+0000","last_became_active":"2023-11-16T23:58:50.740348+0000","last_became_peered":"2023-11-16T23:58:50.740348+0000","last_unstale":"2023-11-16T23:58:51.976155+0000","last_undegraded":"2023-11-16T23:58:51.976155+0000","last_fullsized":"2023-11-16T23:58:51.976155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:59:30.440820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734107+0000","last_change":"2023-11-16T23:58:45.723953+0000","last_active":"2023-11-16T23:59:32.734107+0000","last_peered":"2023-11-16T23:59:32.734107+0000","last_clean":"2023-11-16T23:59:32.734107+0000","last_became_active":"2023-11-16T23:58:45.722075+0000","last_became_peered":"2023-11-16T23:58:45.722075+0000","last_unstale":"2023-11-16T23:59:32.734107+0000","last_undegraded":"2023-11-16T23:59:32.734107+0000","last_fullsized":"2023-11-16T23:59:32.734107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:46:29.950815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240548+0000","last_change":"2023-11-16T23:58:46.290530+0000","last_active":"2023-11-16T23:58:51.240548+0000","last_peered":"2023-11-16T23:58:51.240548+0000","last_clean":"2023-11-16T23:58:51.240548+0000","last_became_active":"2023-11-16T23:58:46.290181+0000","last_became_peered":"2023-11-16T23:58:46.290181+0000","last_unstale":"2023-11-16T23:58:51.240548+0000","last_undegraded":"2023-11-16T23:58:51.240548+0000","last_fullsized":"2023-11-16T23:58:51.240548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:20.504586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974154+0000","last_change":"2023-11-16T23:58:50.739559+0000","last_active":"2023-11-16T23:58:51.974154+0000","last_peered":"2023-11-16T23:58:51.974154+0000","last_clean":"2023-11-16T23:58:51.974154+0000","last_became_active":"2023-11-16T23:58:50.739156+0000","last_became_peered":"2023-11-16T23:58:50.739156+0000","last_unstale":"2023-11-16T23:58:51.974154+0000","last_undegraded":"2023-11-16T23:58:51.974154+0000","last_fullsized":"2023-11-16T23:58:51.974154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:45.312325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416423+0000","last_change":"2023-11-16T23:58:45.723679+0000","last_active":"2023-11-16T23:58:51.416423+0000","last_peered":"2023-11-16T23:58:51.416423+0000","last_clean":"2023-11-16T23:58:51.416423+0000","last_became_active":"2023-11-16T23:58:45.723340+0000","last_became_peered":"2023-11-16T23:58:45.723340+0000","last_unstale":"2023-11-16T23:58:51.416423+0000","last_undegraded":"2023-11-16T23:58:51.416423+0000","last_fullsized":"2023-11-16T23:58:51.416423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:28:17.612030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974645+0000","last_change":"2023-11-16T23:58:46.297004+0000","last_active":"2023-11-16T23:58:51.974645+0000","last_peered":"2023-11-16T23:58:51.974645+0000","last_clean":"2023-11-16T23:58:51.974645+0000","last_became_active":"2023-11-16T23:58:46.296622+0000","last_became_peered":"2023-11-16T23:58:46.296622+0000","last_unstale":"2023-11-16T23:58:51.974645+0000","last_undegraded":"2023-11-16T23:58:51.974645+0000","last_fullsized":"2023-11-16T23:58:51.974645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:00:00.033560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974484+0000","last_change":"2023-11-16T23:58:50.927313+0000","last_active":"2023-11-16T23:58:51.974484+0000","last_peered":"2023-11-16T23:58:51.974484+0000","last_clean":"2023-11-16T23:58:51.974484+0000","last_became_active":"2023-11-16T23:58:50.926865+0000","last_became_peered":"2023-11-16T23:58:50.926865+0000","last_unstale":"2023-11-16T23:58:51.974484+0000","last_undegraded":"2023-11-16T23:58:51.974484+0000","last_fullsized":"2023-11-16T23:58:51.974484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:41:57.718515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733211+0000","last_change":"2023-11-16T23:58:45.714188+0000","last_active":"2023-11-16T23:59:32.733211+0000","last_peered":"2023-11-16T23:59:32.733211+0000","last_clean":"2023-11-16T23:59:32.733211+0000","last_became_active":"2023-11-16T23:58:45.713785+0000","last_became_peered":"2023-11-16T23:58:45.713785+0000","last_unstale":"2023-11-16T23:59:32.733211+0000","last_undegraded":"2023-11-16T23:59:32.733211+0000","last_fullsized":"2023-11-16T23:59:32.733211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:49:36.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239324+0000","last_change":"2023-11-16T23:58:46.291307+0000","last_active":"2023-11-16T23:58:51.239324+0000","last_peered":"2023-11-16T23:58:51.239324+0000","last_clean":"2023-11-16T23:58:51.239324+0000","last_became_active":"2023-11-16T23:58:46.289946+0000","last_became_peered":"2023-11-16T23:58:46.289946+0000","last_unstale":"2023-11-16T23:58:51.239324+0000","last_undegraded":"2023-11-16T23:58:51.239324+0000","last_fullsized":"2023-11-16T23:58:51.239324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:58:49.392838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973947+0000","last_change":"2023-11-16T23:58:50.737205+0000","last_active":"2023-11-16T23:58:51.973947+0000","last_peered":"2023-11-16T23:58:51.973947+0000","last_clean":"2023-11-16T23:58:51.973947+0000","last_became_active":"2023-11-16T23:58:50.736756+0000","last_became_peered":"2023-11-16T23:58:50.736756+0000","last_unstale":"2023-11-16T23:58:51.973947+0000","last_undegraded":"2023-11-16T23:58:51.973947+0000","last_fullsized":"2023-11-16T23:58:51.973947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:05:08.988766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416782+0000","last_change":"2023-11-16T23:58:45.713979+0000","last_active":"2023-11-16T23:58:51.416782+0000","last_peered":"2023-11-16T23:58:51.416782+0000","last_clean":"2023-11-16T23:58:51.416782+0000","last_became_active":"2023-11-16T23:58:45.713492+0000","last_became_peered":"2023-11-16T23:58:45.713492+0000","last_unstale":"2023-11-16T23:58:51.416782+0000","last_undegraded":"2023-11-16T23:58:51.416782+0000","last_fullsized":"2023-11-16T23:58:51.416782+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:17:15.481986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239735+0000","last_change":"2023-11-16T23:58:46.289100+0000","last_active":"2023-11-16T23:58:51.239735+0000","last_peered":"2023-11-16T23:58:51.239735+0000","last_clean":"2023-11-16T23:58:51.239735+0000","last_became_active":"2023-11-16T23:58:46.288506+0000","last_became_peered":"2023-11-16T23:58:46.288506+0000","last_unstale":"2023-11-16T23:58:51.239735+0000","last_undegraded":"2023-11-16T23:58:51.239735+0000","last_fullsized":"2023-11-16T23:58:51.239735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:44:12.486352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242604+0000","last_change":"2023-11-16T23:58:50.741205+0000","last_active":"2023-11-16T23:58:51.242604+0000","last_peered":"2023-11-16T23:58:51.242604+0000","last_clean":"2023-11-16T23:58:51.242604+0000","last_became_active":"2023-11-16T23:58:50.740871+0000","last_became_peered":"2023-11-16T23:58:50.740871+0000","last_unstale":"2023-11-16T23:58:51.242604+0000","last_undegraded":"2023-11-16T23:58:51.242604+0000","last_fullsized":"2023-11-16T23:58:51.242604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:25:17.203133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584250+0000","last_change":"2023-11-16T23:58:45.723618+0000","last_active":"2023-11-16T23:58:51.584250+0000","last_peered":"2023-11-16T23:58:51.584250+0000","last_clean":"2023-11-16T23:58:51.584250+0000","last_became_active":"2023-11-16T23:58:45.723266+0000","last_became_peered":"2023-11-16T23:58:45.723266+0000","last_unstale":"2023-11-16T23:58:51.584250+0000","last_undegraded":"2023-11-16T23:58:51.584250+0000","last_fullsized":"2023-11-16T23:58:51.584250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:20:44.206104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239349+0000","last_change":"2023-11-16T23:58:46.295295+0000","last_active":"2023-11-16T23:58:51.239349+0000","last_peered":"2023-11-16T23:58:51.239349+0000","last_clean":"2023-11-16T23:58:51.239349+0000","last_became_active":"2023-11-16T23:58:46.294801+0000","last_became_peered":"2023-11-16T23:58:46.294801+0000","last_unstale":"2023-11-16T23:58:51.239349+0000","last_undegraded":"2023-11-16T23:58:51.239349+0000","last_fullsized":"2023-11-16T23:58:51.239349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:00.523212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583352+0000","last_change":"2023-11-16T23:58:50.725856+0000","last_active":"2023-11-16T23:58:51.583352+0000","last_peered":"2023-11-16T23:58:51.583352+0000","last_clean":"2023-11-16T23:58:51.583352+0000","last_became_active":"2023-11-16T23:58:50.725453+0000","last_became_peered":"2023-11-16T23:58:50.725453+0000","last_unstale":"2023-11-16T23:58:51.583352+0000","last_undegraded":"2023-11-16T23:58:51.583352+0000","last_fullsized":"2023-11-16T23:58:51.583352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:32:17.547808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976506+0000","last_change":"2023-11-16T23:58:45.718226+0000","last_active":"2023-11-16T23:58:51.976506+0000","last_peered":"2023-11-16T23:58:51.976506+0000","last_clean":"2023-11-16T23:58:51.976506+0000","last_became_active":"2023-11-16T23:58:45.717893+0000","last_became_peered":"2023-11-16T23:58:45.717893+0000","last_unstale":"2023-11-16T23:58:51.976506+0000","last_undegraded":"2023-11-16T23:58:51.976506+0000","last_fullsized":"2023-11-16T23:58:51.976506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:01.114364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239740+0000","last_change":"2023-11-16T23:58:46.289906+0000","last_active":"2023-11-16T23:58:51.239740+0000","last_peered":"2023-11-16T23:58:51.239740+0000","last_clean":"2023-11-16T23:58:51.239740+0000","last_became_active":"2023-11-16T23:58:46.289266+0000","last_became_peered":"2023-11-16T23:58:46.289266+0000","last_unstale":"2023-11-16T23:58:51.239740+0000","last_undegraded":"2023-11-16T23:58:51.239740+0000","last_fullsized":"2023-11-16T23:58:51.239740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:54:03.586113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240620+0000","last_change":"2023-11-16T23:58:46.292220+0000","last_active":"2023-11-16T23:58:51.240620+0000","last_peered":"2023-11-16T23:58:51.240620+0000","last_clean":"2023-11-16T23:58:51.240620+0000","last_became_active":"2023-11-16T23:58:46.291840+0000","last_became_peered":"2023-11-16T23:58:46.291840+0000","last_unstale":"2023-11-16T23:58:51.240620+0000","last_undegraded":"2023-11-16T23:58:51.240620+0000","last_fullsized":"2023-11-16T23:58:51.240620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:18:35.280653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416702+0000","last_change":"2023-11-16T23:58:46.287601+0000","last_active":"2023-11-16T23:58:51.416702+0000","last_peered":"2023-11-16T23:58:51.416702+0000","last_clean":"2023-11-16T23:58:51.416702+0000","last_became_active":"2023-11-16T23:58:46.284738+0000","last_became_peered":"2023-11-16T23:58:46.284738+0000","last_unstale":"2023-11-16T23:58:51.416702+0000","last_undegraded":"2023-11-16T23:58:51.416702+0000","last_fullsized":"2023-11-16T23:58:51.416702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:49.913458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416384+0000","last_change":"2023-11-16T23:58:46.288601+0000","last_active":"2023-11-16T23:58:51.416384+0000","last_peered":"2023-11-16T23:58:51.416384+0000","last_clean":"2023-11-16T23:58:51.416384+0000","last_became_active":"2023-11-16T23:58:46.288266+0000","last_became_peered":"2023-11-16T23:58:46.288266+0000","last_unstale":"2023-11-16T23:58:51.416384+0000","last_undegraded":"2023-11-16T23:58:51.416384+0000","last_fullsized":"2023-11-16T23:58:51.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:18.659483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242778+0000","last_change":"2023-11-16T23:58:46.286495+0000","last_active":"2023-11-16T23:58:51.242778+0000","last_peered":"2023-11-16T23:58:51.242778+0000","last_clean":"2023-11-16T23:58:51.242778+0000","last_became_active":"2023-11-16T23:58:46.281157+0000","last_became_peered":"2023-11-16T23:58:46.281157+0000","last_unstale":"2023-11-16T23:58:51.242778+0000","last_undegraded":"2023-11-16T23:58:51.242778+0000","last_fullsized":"2023-11-16T23:58:51.242778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:02.565515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976001+0000","last_change":"2023-11-16T23:58:46.282373+0000","last_active":"2023-11-16T23:58:51.976001+0000","last_peered":"2023-11-16T23:58:51.976001+0000","last_clean":"2023-11-16T23:58:51.976001+0000","last_became_active":"2023-11-16T23:58:46.282028+0000","last_became_peered":"2023-11-16T23:58:46.282028+0000","last_unstale":"2023-11-16T23:58:51.976001+0000","last_undegraded":"2023-11-16T23:58:51.976001+0000","last_fullsized":"2023-11-16T23:58:51.976001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:29:48.640544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241073+0000","last_change":"2023-11-16T23:58:46.285528+0000","last_active":"2023-11-16T23:58:51.241073+0000","last_peered":"2023-11-16T23:58:51.241073+0000","last_clean":"2023-11-16T23:58:51.241073+0000","last_became_active":"2023-11-16T23:58:46.285227+0000","last_became_peered":"2023-11-16T23:58:46.285227+0000","last_unstale":"2023-11-16T23:58:51.241073+0000","last_undegraded":"2023-11-16T23:58:51.241073+0000","last_fullsized":"2023-11-16T23:58:51.241073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:45:11.259431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416982+0000","last_change":"2023-11-16T23:58:46.283514+0000","last_active":"2023-11-16T23:58:51.416982+0000","last_peered":"2023-11-16T23:58:51.416982+0000","last_clean":"2023-11-16T23:58:51.416982+0000","last_became_active":"2023-11-16T23:58:46.283026+0000","last_became_peered":"2023-11-16T23:58:46.283026+0000","last_unstale":"2023-11-16T23:58:51.416982+0000","last_undegraded":"2023-11-16T23:58:51.416982+0000","last_fullsized":"2023-11-16T23:58:51.416982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:14.255094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974891+0000","last_change":"2023-11-16T23:58:46.283334+0000","last_active":"2023-11-16T23:58:51.974891+0000","last_peered":"2023-11-16T23:58:51.974891+0000","last_clean":"2023-11-16T23:58:51.974891+0000","last_became_active":"2023-11-16T23:58:46.283012+0000","last_became_peered":"2023-11-16T23:58:46.283012+0000","last_unstale":"2023-11-16T23:58:51.974891+0000","last_undegraded":"2023-11-16T23:58:51.974891+0000","last_fullsized":"2023-11-16T23:58:51.974891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:44.161087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583457+0000","last_change":"2023-11-16T23:58:46.292720+0000","last_active":"2023-11-16T23:58:51.583457+0000","last_peered":"2023-11-16T23:58:51.583457+0000","last_clean":"2023-11-16T23:58:51.583457+0000","last_became_active":"2023-11-16T23:58:46.292109+0000","last_became_peered":"2023-11-16T23:58:46.292109+0000","last_unstale":"2023-11-16T23:58:51.583457+0000","last_undegraded":"2023-11-16T23:58:51.583457+0000","last_fullsized":"2023-11-16T23:58:51.583457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:41:12.067371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974297+0000","last_change":"2023-11-16T23:58:46.279919+0000","last_active":"2023-11-16T23:58:51.974297+0000","last_peered":"2023-11-16T23:58:51.974297+0000","last_clean":"2023-11-16T23:58:51.974297+0000","last_became_active":"2023-11-16T23:58:46.279601+0000","last_became_peered":"2023-11-16T23:58:46.279601+0000","last_unstale":"2023-11-16T23:58:51.974297+0000","last_undegraded":"2023-11-16T23:58:51.974297+0000","last_fullsized":"2023-11-16T23:58:51.974297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:28:22.080722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975063+0000","last_change":"2023-11-16T23:58:46.293091+0000","last_active":"2023-11-16T23:58:51.975063+0000","last_peered":"2023-11-16T23:58:51.975063+0000","last_clean":"2023-11-16T23:58:51.975063+0000","last_became_active":"2023-11-16T23:58:46.292728+0000","last_became_peered":"2023-11-16T23:58:46.292728+0000","last_unstale":"2023-11-16T23:58:51.975063+0000","last_undegraded":"2023-11-16T23:58:51.975063+0000","last_fullsized":"2023-11-16T23:58:51.975063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:57:24.095879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975716+0000","last_change":"2023-11-16T23:58:46.281708+0000","last_active":"2023-11-16T23:58:51.975716+0000","last_peered":"2023-11-16T23:58:51.975716+0000","last_clean":"2023-11-16T23:58:51.975716+0000","last_became_active":"2023-11-16T23:58:46.281335+0000","last_became_peered":"2023-11-16T23:58:46.281335+0000","last_unstale":"2023-11-16T23:58:51.975716+0000","last_undegraded":"2023-11-16T23:58:51.975716+0000","last_fullsized":"2023-11-16T23:58:51.975716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:46:24.208579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240805+0000","last_change":"2023-11-16T23:58:46.286046+0000","last_active":"2023-11-16T23:58:51.240805+0000","last_peered":"2023-11-16T23:58:51.240805+0000","last_clean":"2023-11-16T23:58:51.240805+0000","last_became_active":"2023-11-16T23:58:46.285702+0000","last_became_peered":"2023-11-16T23:58:46.285702+0000","last_unstale":"2023-11-16T23:58:51.240805+0000","last_undegraded":"2023-11-16T23:58:51.240805+0000","last_fullsized":"2023-11-16T23:58:51.240805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:15:25.406768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":116,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":54053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":79,"seq":339302416405,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":974848,"data_stored":475644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85599999999999998}]},{"osd":7,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.96099999999999997}]},{"osd":10,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.81999999999999995}]}]},{"osd":10,"up_from":71,"seq":304942678041,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738212,"statfs":{"total":95995035648,"available":95987929088,"internally_reserved":0,"allocated":1593344,"data_stored":1065753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.31}]},{"osd":2,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.39600000000000002}]},{"osd":5,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.434},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67800000000000005}]},{"osd":9,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.64700000000000002}]},{"osd":11,"last update":"Thu Nov 16 23:58:58 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.63400000000000001}]}]},{"osd":9,"up_from":66,"seq":283467841570,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6292,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738860,"statfs":{"total":95995035648,"available":95988592640,"internally_reserved":0,"allocated":995328,"data_stored":479976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.434}]},{"osd":1,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.79600000000000004,"15min":0.81200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0669999999999999,"5min":1.139,"15min":1.139},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.97899999999999998,"5min":1.044,"15min":1.044},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.83799999999999997,"15min":0.85599999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.85199999999999998,"15min":0.872},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.03,"5min":1.216,"15min":1.216},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.92900000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0429999999999999,"5min":1.329,"15min":1.329},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.877,"5min":0.88500000000000001,"15min":0.88700000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Nov 16 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.86299999999999999}]},{"osd":7,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.97099999999999997,"15min":0.98299999999999998},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.02,"15min":1.038},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.83499999999999996}]},{"osd":8,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.006,"15min":1.0349999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.357,"15min":1.357},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98799999999999999,"15min":0.98999999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.6429999999999998,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.874,"15min":0.878},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0649999999999999,"5min":1.141,"15min":1.141},"last":0.86899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103206,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":978944,"data_stored":471592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.58099999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.748,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89600000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.63400000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.46899999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68799999999999994,"15min":0.67000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":4.5149999999999997,"5min":4.5149999999999997,"15min":4.5149999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84599999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.77700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.91700000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.43099999999999999}]},{"osd":6,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78500000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76500000000000001,"15min":0.75},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.74199999999999999}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77700000000000002,"15min":0.76900000000000002},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":0.85299999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7040,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738112,"statfs":{"total":95995035648,"available":95987826688,"internally_reserved":0,"allocated":1564672,"data_stored":1061916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.53600000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81599999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.66400000000000003}]},{"osd":3,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57299999999999995,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77800000000000002,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.94599999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.501,"15min":0.501},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.499},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.503,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.44400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.45900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.83799999999999997}]},{"osd":6,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":1.069,"15min":1.069},"last":0.64300000000000002}]},{"osd":8,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.626,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.51200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.77800000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.55500000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.87},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.93700000000000006}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6396,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738756,"statfs":{"total":95995035648,"available":95988486144,"internally_reserved":0,"allocated":970752,"data_stored":471056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.35399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86299999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79000000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69399999999999995,"15min":0.68100000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91000000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80200000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.626}]},{"osd":4,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.75800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.111,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73099999999999998,"15min":0.73499999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.85899999999999999,"5min":1.044,"15min":1.044},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.78100000000000003,"15min":0.78900000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.95099999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.754,"15min":0.755},"min":{"1min":0.52300000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94999999999999996,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.81599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78800000000000003,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85899999999999999,"15min":0.86599999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.032,"5min":1.175,"15min":1.175},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.84399999999999997,"15min":0.81200000000000006},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.5839999999999996,"5min":5.5839999999999996,"15min":5.5839999999999996},"last":0.76400000000000001}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.91500000000000004,"15min":0.92500000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.91100000000000003}]},{"osd":10,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.87},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.70599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7384,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737768,"statfs":{"total":95995035648,"available":95987474432,"internally_reserved":0,"allocated":999424,"data_stored":492141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.50700000000000001},"min":{"1min":0.48899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89200000000000002,"5min":0.998,"15min":0.998},"last":0.63},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80700000000000005,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Nov 16 23:59:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.56399999999999995,"15min":0.54400000000000004},"min":{"1min":0.42099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.45200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.41399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78200000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.58899999999999997,"15min":0.55100000000000005},"min":{"1min":0.45900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86799999999999999,"5min":1.028,"15min":1.028},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.496},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.432,"15min":0.40100000000000002},"min":{"1min":0.34300000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.443}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.55600000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.877,"5min":1.103,"15min":1.103},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.38,"15min":0.38},"max":{"1min":0.83099999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.745}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57999999999999996,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.81799999999999995}]},{"osd":8,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.59799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66000000000000003,"15min":0.64000000000000001},"min":{"1min":0.59299999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71599999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73899999999999999,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.96599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79400000000000004,"15min":0.78900000000000003},"min":{"1min":0.625,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.90300000000000002}]},{"osd":10,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.875},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.77300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7364,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737788,"statfs":{"total":95995035648,"available":95987494912,"internally_reserved":0,"allocated":978944,"data_stored":471660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 23:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57999999999999996,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85999999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59599999999999997,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82599999999999996,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.55900000000000005},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64100000000000001,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0169999999999999,"5min":1.071,"15min":1.071},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.51900000000000002,"15min":0.503},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.499},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.35699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.68100000000000005,"15min":0.625},"min":{"1min":0.47699999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.754},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 23:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.61199999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67200000000000004,"15min":0.65500000000000003},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:59:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67900000000000005,"15min":0.67700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.96299999999999997,"5min":1.254,"15min":1.254},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.73599999999999999,"15min":0.70899999999999996},"min":{"1min":0.45900000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.876}]},{"osd":7,"last update":"Thu Nov 16 23:58:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.093,"5min":1.135,"15min":1.135},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.85399999999999998,"15min":0.86499999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.80900000000000005,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1279999999999999,"5min":1.276,"15min":1.276},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.93000000000000005,"15min":0.93200000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.98099999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.042}]},{"osd":11,"last update":"Thu Nov 16 23:59:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.018}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7368,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737784,"statfs":{"total":95995035648,"available":95987490816,"internally_reserved":0,"allocated":983040,"data_stored":474794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.72399999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.50600000000000001},"min":{"1min":0.38200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.42499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.435,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.752,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.72099999999999997}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Nov 16 23:58:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.63},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73299999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.78200000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.749}]},{"osd":8,"last update":"Thu Nov 16 23:58:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64600000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.67500000000000004}]},{"osd":10,"last update":"Thu Nov 16 23:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":470919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.40899999999999997,"15min":0.38400000000000001},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.42599999999999999,"15min":0.40000000000000002},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57599999999999996,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.625,"15min":0.58899999999999997},"min":{"1min":0.55800000000000005,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.69499999999999995,"15min":0.61499999999999999},"min":{"1min":0.499,"5min":0.37,"15min":0.37},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Thu Nov 16 23:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.622},"min":{"1min":0.50900000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.042,"5min":1.405,"15min":1.405},"last":0.87},{"interface":"front","average":{"1min":0.747,"5min":0.65800000000000003,"15min":0.61899999999999999},"min":{"1min":0.54500000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Nov 16 23:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69599999999999995,"15min":0.63900000000000001},"min":{"1min":0.51400000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.1220000000000001,"5min":1.492,"15min":1.492},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68999999999999995,"15min":0.64600000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.212,"15min":1.212},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Nov 16 23:58:55 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58999999999999997,"15min":0.56999999999999995},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.72499999999999998,"15min":0.68899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.98599999999999999}]},{"osd":7,"last update":"Thu Nov 16 23:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.79800000000000004},"min":{"1min":0.66300000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1739999999999999,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77600000000000002,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1419999999999999,"5min":1.256,"15min":1.256},"last":0.92100000000000004}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.90600000000000003,"15min":0.86799999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":7.173,"5min":7.173,"15min":7.173},"last":0.84499999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81599999999999995,"15min":0.82899999999999996},"min":{"1min":0.55900000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2030000000000001,"5min":1.23,"15min":1.23},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.998,"15min":1.014},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.1399999999999999,"5min":1.617,"15min":1.617},"last":0.95899999999999996}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":7.1600000000000001,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":1.036},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.97399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6532,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738620,"statfs":{"total":95995035648,"available":95988346880,"internally_reserved":0,"allocated":978944,"data_stored":471562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.27700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65200000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38600000000000001,"15min":0.36699999999999999},"min":{"1min":0.29199999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56299999999999994,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.379,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.753,"5min":0.872,"15min":0.872},"last":0.62}]},{"osd":3,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.58999999999999997},"min":{"1min":0.53900000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64800000000000002,"15min":0.62},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":1.105,"15min":1.105},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Nov 16 23:59:31 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Nov 16 23:58:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.62,"15min":0.59999999999999998},"min":{"1min":0.46200000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80800000000000005,"5min":0.997,"15min":0.997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.45600000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.71899999999999997}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84399999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.755},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82699999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81899999999999995}]},{"osd":9,"last update":"Thu Nov 16 23:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72299999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.91000000000000003,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.56799999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.90400000000000003,"5min":1.05,"15min":1.05},"last":0.79100000000000004}]},{"osd":10,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.872},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.78000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7188,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737964,"statfs":{"total":95995035648,"available":95987675136,"internally_reserved":0,"allocated":1585152,"data_stored":1062431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52000000000000002},"min":{"1min":0.42199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.753},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.753,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40400000000000003,"15min":0.38800000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55700000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.38700000000000001},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.442}]},{"osd":3,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.503},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.47099999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.55300000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81299999999999994,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.44,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.623}]},{"osd":6,"last update":"Thu Nov 16 23:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63100000000000001,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55700000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63700000000000001,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.746},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.44500000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60199999999999998}]},{"osd":8,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82499999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.58399999999999996}]},{"osd":9,"last update":"Thu Nov 16 23:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.55500000000000005,"5min":0.497,"15min":0.497},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.92900000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68500000000000005}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T23:59:38.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T23:59:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 -- 172.21.15.23:0/4209697878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed181036b0 msgr2=0x7fed18105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 --2- 172.21.15.23:0/4209697878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed181036b0 0x7fed18105a90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed08009990 tx=0x7fed08009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 -- 172.21.15.23:0/4209697878 shutdown_connections 2023-11-16T23:59:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 --2- 172.21.15.23:0/4209697878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed18105fd0 0x7fed181083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 --2- 172.21.15.23:0/4209697878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18100d90 0x7fed18103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 --2- 172.21.15.23:0/4209697878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed181036b0 0x7fed18105a90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 -- 172.21.15.23:0/4209697878 >> 172.21.15.23:0/4209697878 conn(0x7fed180fa750 msgr2=0x7fed180fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.443+0000 7fed1fdff700 1 -- 172.21.15.23:0/4209697878 shutdown_connections 2023-11-16T23:59:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.444+0000 7fed1fdff700 1 -- 172.21.15.23:0/4209697878 wait complete. 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 Processor -- start 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 -- start start 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed181036b0 0x7fed18198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18105fd0 0x7fed1819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:38.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819fd40 con 0x7fed18100d90 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819fe80 con 0x7fed18105fd0 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1fdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed181a0010 con 0x7fed181036b0 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1db9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1db9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52956/0 (socket says 172.21.15.23:52956) 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1db9b700 1 -- 172.21.15.23:0/2444435837 learned_addr learned my addr 172.21.15.23:0/2444435837 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1e39c700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18105fd0 0x7fed1819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.445+0000 7fed1d39a700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed181036b0 0x7fed18198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 -- 172.21.15.23:0/2444435837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed181036b0 msgr2=0x7fed18198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed181036b0 0x7fed18198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 -- 172.21.15.23:0/2444435837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18105fd0 msgr2=0x7fed1819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18105fd0 0x7fed1819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 -- 172.21.15.23:0/2444435837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080095f0 con 0x7fed18100d90 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1db9b700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e970 tx=0x7fed1400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14009790 con 0x7fed18100d90 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed1400c7e0 con 0x7fed18100d90 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1810bc60 con 0x7fed18100d90 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.446+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed140054e0 con 0x7fed18100d90 2023-11-16T23:59:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.447+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1810bad0 con 0x7fed18100d90 2023-11-16T23:59:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.448+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed18107b40 con 0x7fed18100d90 2023-11-16T23:59:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.449+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fed1400c950 con 0x7fed18100d90 2023-11-16T23:59:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.451+0000 7fed0effd700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.451+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fed14014070 con 0x7fed18100d90 2023-11-16T23:59:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.451+0000 7fed1d39a700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.452+0000 7fed1d39a700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 0x7fed040a3710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b2e0 tx=0x7fed08005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.454+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed14096290 con 0x7fed18100d90 2023-11-16T23:59:38.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.659+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed18061c40 con 0x7fed040a1260 2023-11-16T23:59:38.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.672+0000 7fed0effd700 1 -- 172.21.15.23:0/2444435837 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422658 (secure 0 0 0) 0x7fed18061c40 con 0x7fed040a1260 2023-11-16T23:59:38.689 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:38.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 msgr2=0x7fed040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 0x7fed040a3710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b2e0 tx=0x7fed08005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 msgr2=0x7fed181981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:38.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e970 tx=0x7fed1400ec80 comp rx=0 tx=0).stop 2023-11-16T23:59:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 shutdown_connections 2023-11-16T23:59:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed18105fd0 0x7fed1819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100d90 0x7fed181981c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 --2- 172.21.15.23:0/2444435837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed181036b0 0x7fed18198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 >> 172.21.15.23:0/2444435837 conn(0x7fed180fa750 msgr2=0x7fed180ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 shutdown_connections 2023-11-16T23:59:38.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:38.677+0000 7fed1fdff700 1 -- 172.21.15.23:0/2444435837 wait complete. 2023-11-16T23:59:38.705 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-11-16T23:59:38.706 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2023-11-16T23:59:37.064960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597217,"num_objects":56,"num_object_clones":0,"num_object_copies":169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":145,"num_read_kb":128,"num_write":298,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81648,"kb_used_data":13264,"kb_used_omap":0,"kb_used_meta":68288,"kb_avail":1124860176,"statfs":{"total":1151940427776,"available":1151856820224,"internally_reserved":0,"allocated":13582336,"data_stored":7469444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69926912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2257,"num_objects":32,"num_object_clones":0,"num_object_copies":97,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":68,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003737"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241211+0000","last_change":"2023-11-16T23:58:46.280372+0000","last_active":"2023-11-16T23:58:51.241211+0000","last_peered":"2023-11-16T23:58:51.241211+0000","last_clean":"2023-11-16T23:58:51.241211+0000","last_became_active":"2023-11-16T23:58:46.280021+0000","last_became_peered":"2023-11-16T23:58:46.280021+0000","last_unstale":"2023-11-16T23:58:51.241211+0000","last_undegraded":"2023-11-16T23:58:51.241211+0000","last_fullsized":"2023-11-16T23:58:51.241211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:37.900152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243114+0000","last_change":"2023-11-16T23:58:46.297046+0000","last_active":"2023-11-16T23:58:51.243114+0000","last_peered":"2023-11-16T23:58:51.243114+0000","last_clean":"2023-11-16T23:58:51.243114+0000","last_became_active":"2023-11-16T23:58:46.282128+0000","last_became_peered":"2023-11-16T23:58:46.282128+0000","last_unstale":"2023-11-16T23:58:51.243114+0000","last_undegraded":"2023-11-16T23:58:51.243114+0000","last_fullsized":"2023-11-16T23:58:51.243114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:56.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242939+0000","last_change":"2023-11-16T23:58:46.283752+0000","last_active":"2023-11-16T23:58:51.242939+0000","last_peered":"2023-11-16T23:58:51.242939+0000","last_clean":"2023-11-16T23:58:51.242939+0000","last_became_active":"2023-11-16T23:58:46.280787+0000","last_became_peered":"2023-11-16T23:58:46.280787+0000","last_unstale":"2023-11-16T23:58:51.242939+0000","last_undegraded":"2023-11-16T23:58:51.242939+0000","last_fullsized":"2023-11-16T23:58:51.242939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:06.203820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974468+0000","last_change":"2023-11-16T23:58:46.282847+0000","last_active":"2023-11-16T23:58:51.974468+0000","last_peered":"2023-11-16T23:58:51.974468+0000","last_clean":"2023-11-16T23:58:51.974468+0000","last_became_active":"2023-11-16T23:58:46.282527+0000","last_became_peered":"2023-11-16T23:58:46.282527+0000","last_unstale":"2023-11-16T23:58:51.974468+0000","last_undegraded":"2023-11-16T23:58:51.974468+0000","last_fullsized":"2023-11-16T23:58:51.974468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:15.327419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973484+0000","last_change":"2023-11-16T23:58:46.288154+0000","last_active":"2023-11-16T23:58:51.973484+0000","last_peered":"2023-11-16T23:58:51.973484+0000","last_clean":"2023-11-16T23:58:51.973484+0000","last_became_active":"2023-11-16T23:58:46.287752+0000","last_became_peered":"2023-11-16T23:58:46.287752+0000","last_unstale":"2023-11-16T23:58:51.973484+0000","last_undegraded":"2023-11-16T23:58:51.973484+0000","last_fullsized":"2023-11-16T23:58:51.973484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:17.777171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975139+0000","last_change":"2023-11-16T23:58:46.283177+0000","last_active":"2023-11-16T23:58:51.975139+0000","last_peered":"2023-11-16T23:58:51.975139+0000","last_clean":"2023-11-16T23:58:51.975139+0000","last_became_active":"2023-11-16T23:58:46.282753+0000","last_became_peered":"2023-11-16T23:58:46.282753+0000","last_unstale":"2023-11-16T23:58:51.975139+0000","last_undegraded":"2023-11-16T23:58:51.975139+0000","last_fullsized":"2023-11-16T23:58:51.975139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:40.545830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238882+0000","last_change":"2023-11-16T23:58:46.296002+0000","last_active":"2023-11-16T23:58:51.238882+0000","last_peered":"2023-11-16T23:58:51.238882+0000","last_clean":"2023-11-16T23:58:51.238882+0000","last_became_active":"2023-11-16T23:58:46.295191+0000","last_became_peered":"2023-11-16T23:58:46.295191+0000","last_unstale":"2023-11-16T23:58:51.238882+0000","last_undegraded":"2023-11-16T23:58:51.238882+0000","last_fullsized":"2023-11-16T23:58:51.238882+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:32:43.802637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973904+0000","last_change":"2023-11-16T23:58:46.294824+0000","last_active":"2023-11-16T23:58:51.973904+0000","last_peered":"2023-11-16T23:58:51.973904+0000","last_clean":"2023-11-16T23:58:51.973904+0000","last_became_active":"2023-11-16T23:58:46.294502+0000","last_became_peered":"2023-11-16T23:58:46.294502+0000","last_unstale":"2023-11-16T23:58:51.973904+0000","last_undegraded":"2023-11-16T23:58:51.973904+0000","last_fullsized":"2023-11-16T23:58:51.973904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:47:28.853625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974608+0000","last_change":"2023-11-16T23:58:46.292583+0000","last_active":"2023-11-16T23:58:51.974608+0000","last_peered":"2023-11-16T23:58:51.974608+0000","last_clean":"2023-11-16T23:58:51.974608+0000","last_became_active":"2023-11-16T23:58:46.292089+0000","last_became_peered":"2023-11-16T23:58:46.292089+0000","last_unstale":"2023-11-16T23:58:51.974608+0000","last_undegraded":"2023-11-16T23:58:51.974608+0000","last_fullsized":"2023-11-16T23:58:51.974608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:54.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975325+0000","last_change":"2023-11-16T23:58:46.289410+0000","last_active":"2023-11-16T23:58:51.975325+0000","last_peered":"2023-11-16T23:58:51.975325+0000","last_clean":"2023-11-16T23:58:51.975325+0000","last_became_active":"2023-11-16T23:58:46.288746+0000","last_became_peered":"2023-11-16T23:58:46.288746+0000","last_unstale":"2023-11-16T23:58:51.975325+0000","last_undegraded":"2023-11-16T23:58:51.975325+0000","last_fullsized":"2023-11-16T23:58:51.975325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:39:36.326325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239625+0000","last_change":"2023-11-16T23:58:46.288862+0000","last_active":"2023-11-16T23:58:51.239625+0000","last_peered":"2023-11-16T23:58:51.239625+0000","last_clean":"2023-11-16T23:58:51.239625+0000","last_became_active":"2023-11-16T23:58:46.288046+0000","last_became_peered":"2023-11-16T23:58:46.288046+0000","last_unstale":"2023-11-16T23:58:51.239625+0000","last_undegraded":"2023-11-16T23:58:51.239625+0000","last_fullsized":"2023-11-16T23:58:51.239625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:28:13.350271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974869+0000","last_change":"2023-11-16T23:58:46.283219+0000","last_active":"2023-11-16T23:58:51.974869+0000","last_peered":"2023-11-16T23:58:51.974869+0000","last_clean":"2023-11-16T23:58:51.974869+0000","last_became_active":"2023-11-16T23:58:46.282759+0000","last_became_peered":"2023-11-16T23:58:46.282759+0000","last_unstale":"2023-11-16T23:58:51.974869+0000","last_undegraded":"2023-11-16T23:58:51.974869+0000","last_fullsized":"2023-11-16T23:58:51.974869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:02:48.010586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:46.292139+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:46.291682+0000","last_became_peered":"2023-11-16T23:58:46.291682+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:27:16.238637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242504+0000","last_change":"2023-11-16T23:58:46.293326+0000","last_active":"2023-11-16T23:58:51.242504+0000","last_peered":"2023-11-16T23:58:51.242504+0000","last_clean":"2023-11-16T23:58:51.242504+0000","last_became_active":"2023-11-16T23:58:46.286619+0000","last_became_peered":"2023-11-16T23:58:46.286619+0000","last_unstale":"2023-11-16T23:58:51.242504+0000","last_undegraded":"2023-11-16T23:58:51.242504+0000","last_fullsized":"2023-11-16T23:58:51.242504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:41.213234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242574+0000","last_change":"2023-11-16T23:58:46.274095+0000","last_active":"2023-11-16T23:58:51.242574+0000","last_peered":"2023-11-16T23:58:51.242574+0000","last_clean":"2023-11-16T23:58:51.242574+0000","last_became_active":"2023-11-16T23:58:46.272718+0000","last_became_peered":"2023-11-16T23:58:46.272718+0000","last_unstale":"2023-11-16T23:58:51.242574+0000","last_undegraded":"2023-11-16T23:58:51.242574+0000","last_fullsized":"2023-11-16T23:58:51.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:45.267014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973636+0000","last_change":"2023-11-16T23:58:46.294079+0000","last_active":"2023-11-16T23:58:51.973636+0000","last_peered":"2023-11-16T23:58:51.973636+0000","last_clean":"2023-11-16T23:58:51.973636+0000","last_became_active":"2023-11-16T23:58:46.293738+0000","last_became_peered":"2023-11-16T23:58:46.293738+0000","last_unstale":"2023-11-16T23:58:51.973636+0000","last_undegraded":"2023-11-16T23:58:51.973636+0000","last_fullsized":"2023-11-16T23:58:51.973636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:48.113990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241160+0000","last_change":"2023-11-16T23:58:46.296439+0000","last_active":"2023-11-16T23:58:51.241160+0000","last_peered":"2023-11-16T23:58:51.241160+0000","last_clean":"2023-11-16T23:58:51.241160+0000","last_became_active":"2023-11-16T23:58:46.296002+0000","last_became_peered":"2023-11-16T23:58:46.296002+0000","last_unstale":"2023-11-16T23:58:51.241160+0000","last_undegraded":"2023-11-16T23:58:51.241160+0000","last_fullsized":"2023-11-16T23:58:51.241160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:45.679213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975910+0000","last_change":"2023-11-16T23:58:46.287374+0000","last_active":"2023-11-16T23:58:51.975910+0000","last_peered":"2023-11-16T23:58:51.975910+0000","last_clean":"2023-11-16T23:58:51.975910+0000","last_became_active":"2023-11-16T23:58:46.287044+0000","last_became_peered":"2023-11-16T23:58:46.287044+0000","last_unstale":"2023-11-16T23:58:51.975910+0000","last_undegraded":"2023-11-16T23:58:51.975910+0000","last_fullsized":"2023-11-16T23:58:51.975910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:31:41.509204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416818+0000","last_change":"2023-11-16T23:58:46.284627+0000","last_active":"2023-11-16T23:58:51.416818+0000","last_peered":"2023-11-16T23:58:51.416818+0000","last_clean":"2023-11-16T23:58:51.416818+0000","last_became_active":"2023-11-16T23:58:46.284221+0000","last_became_peered":"2023-11-16T23:58:46.284221+0000","last_unstale":"2023-11-16T23:58:51.416818+0000","last_undegraded":"2023-11-16T23:58:51.416818+0000","last_fullsized":"2023-11-16T23:58:51.416818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:37.717489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416711+0000","last_change":"2023-11-16T23:58:50.742742+0000","last_active":"2023-11-16T23:58:51.416711+0000","last_peered":"2023-11-16T23:58:51.416711+0000","last_clean":"2023-11-16T23:58:51.416711+0000","last_became_active":"2023-11-16T23:58:50.742337+0000","last_became_peered":"2023-11-16T23:58:50.742337+0000","last_unstale":"2023-11-16T23:58:51.416711+0000","last_undegraded":"2023-11-16T23:58:51.416711+0000","last_fullsized":"2023-11-16T23:58:51.416711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:03:40.742862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717374+0000","last_change":"2023-11-16T23:58:45.714612+0000","last_active":"2023-11-16T23:59:33.717374+0000","last_peered":"2023-11-16T23:59:33.717374+0000","last_clean":"2023-11-16T23:59:33.717374+0000","last_became_active":"2023-11-16T23:58:45.714235+0000","last_became_peered":"2023-11-16T23:58:45.714235+0000","last_unstale":"2023-11-16T23:59:33.717374+0000","last_undegraded":"2023-11-16T23:59:33.717374+0000","last_fullsized":"2023-11-16T23:59:33.717374+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:20:19.287101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974930+0000","last_change":"2023-11-16T23:58:46.294399+0000","last_active":"2023-11-16T23:58:51.974930+0000","last_peered":"2023-11-16T23:58:51.974930+0000","last_clean":"2023-11-16T23:58:51.974930+0000","last_became_active":"2023-11-16T23:58:46.291722+0000","last_became_peered":"2023-11-16T23:58:46.291722+0000","last_unstale":"2023-11-16T23:58:51.974930+0000","last_undegraded":"2023-11-16T23:58:51.974930+0000","last_fullsized":"2023-11-16T23:58:51.974930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:58:25.110448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973733+0000","last_change":"2023-11-16T23:58:50.745764+0000","last_active":"2023-11-16T23:58:51.973733+0000","last_peered":"2023-11-16T23:58:51.973733+0000","last_clean":"2023-11-16T23:58:51.973733+0000","last_became_active":"2023-11-16T23:58:50.745411+0000","last_became_peered":"2023-11-16T23:58:50.745411+0000","last_unstale":"2023-11-16T23:58:51.973733+0000","last_undegraded":"2023-11-16T23:58:51.973733+0000","last_fullsized":"2023-11-16T23:58:51.973733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:04:04.552603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975096+0000","last_change":"2023-11-16T23:58:45.724215+0000","last_active":"2023-11-16T23:58:51.975096+0000","last_peered":"2023-11-16T23:58:51.975096+0000","last_clean":"2023-11-16T23:58:51.975096+0000","last_became_active":"2023-11-16T23:58:45.723866+0000","last_became_peered":"2023-11-16T23:58:45.723866+0000","last_unstale":"2023-11-16T23:58:51.975096+0000","last_undegraded":"2023-11-16T23:58:51.975096+0000","last_fullsized":"2023-11-16T23:58:51.975096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:09.212226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243119+0000","last_change":"2023-11-16T23:58:46.291701+0000","last_active":"2023-11-16T23:58:51.243119+0000","last_peered":"2023-11-16T23:58:51.243119+0000","last_clean":"2023-11-16T23:58:51.243119+0000","last_became_active":"2023-11-16T23:58:46.282612+0000","last_became_peered":"2023-11-16T23:58:46.282612+0000","last_unstale":"2023-11-16T23:58:51.243119+0000","last_undegraded":"2023-11-16T23:58:51.243119+0000","last_fullsized":"2023-11-16T23:58:51.243119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:18.169006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975347+0000","last_change":"2023-11-16T23:58:50.731025+0000","last_active":"2023-11-16T23:58:51.975347+0000","last_peered":"2023-11-16T23:58:51.975347+0000","last_clean":"2023-11-16T23:58:51.975347+0000","last_became_active":"2023-11-16T23:58:50.730593+0000","last_became_peered":"2023-11-16T23:58:50.730593+0000","last_unstale":"2023-11-16T23:58:51.975347+0000","last_undegraded":"2023-11-16T23:58:51.975347+0000","last_fullsized":"2023-11-16T23:58:51.975347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:18.213726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715505+0000","last_change":"2023-11-16T23:58:45.716403+0000","last_active":"2023-11-16T23:59:33.715505+0000","last_peered":"2023-11-16T23:59:33.715505+0000","last_clean":"2023-11-16T23:59:33.715505+0000","last_became_active":"2023-11-16T23:58:45.716050+0000","last_became_peered":"2023-11-16T23:58:45.716050+0000","last_unstale":"2023-11-16T23:59:33.715505+0000","last_undegraded":"2023-11-16T23:59:33.715505+0000","last_fullsized":"2023-11-16T23:59:33.715505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:42:55.310961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974544+0000","last_change":"2023-11-16T23:58:46.286460+0000","last_active":"2023-11-16T23:58:51.974544+0000","last_peered":"2023-11-16T23:58:51.974544+0000","last_clean":"2023-11-16T23:58:51.974544+0000","last_became_active":"2023-11-16T23:58:46.286135+0000","last_became_peered":"2023-11-16T23:58:46.286135+0000","last_unstale":"2023-11-16T23:58:51.974544+0000","last_undegraded":"2023-11-16T23:58:51.974544+0000","last_fullsized":"2023-11-16T23:58:51.974544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:15:02.309346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973789+0000","last_change":"2023-11-16T23:58:50.737134+0000","last_active":"2023-11-16T23:58:51.973789+0000","last_peered":"2023-11-16T23:58:51.973789+0000","last_clean":"2023-11-16T23:58:51.973789+0000","last_became_active":"2023-11-16T23:58:50.736489+0000","last_became_peered":"2023-11-16T23:58:50.736489+0000","last_unstale":"2023-11-16T23:58:51.973789+0000","last_undegraded":"2023-11-16T23:58:51.973789+0000","last_fullsized":"2023-11-16T23:58:51.973789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:33.961942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973818+0000","last_change":"2023-11-16T23:58:45.713153+0000","last_active":"2023-11-16T23:58:51.973818+0000","last_peered":"2023-11-16T23:58:51.973818+0000","last_clean":"2023-11-16T23:58:51.973818+0000","last_became_active":"2023-11-16T23:58:45.712626+0000","last_became_peered":"2023-11-16T23:58:45.712626+0000","last_unstale":"2023-11-16T23:58:51.973818+0000","last_undegraded":"2023-11-16T23:58:51.973818+0000","last_fullsized":"2023-11-16T23:58:51.973818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:55:36.907838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584145+0000","last_change":"2023-11-16T23:58:46.289729+0000","last_active":"2023-11-16T23:58:51.584145+0000","last_peered":"2023-11-16T23:58:51.584145+0000","last_clean":"2023-11-16T23:58:51.584145+0000","last_became_active":"2023-11-16T23:58:46.284791+0000","last_became_peered":"2023-11-16T23:58:46.284791+0000","last_unstale":"2023-11-16T23:58:51.584145+0000","last_undegraded":"2023-11-16T23:58:51.584145+0000","last_fullsized":"2023-11-16T23:58:51.584145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:13:27.839983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416962+0000","last_change":"2023-11-16T23:58:50.725585+0000","last_active":"2023-11-16T23:58:51.416962+0000","last_peered":"2023-11-16T23:58:51.416962+0000","last_clean":"2023-11-16T23:58:51.416962+0000","last_became_active":"2023-11-16T23:58:50.725188+0000","last_became_peered":"2023-11-16T23:58:50.725188+0000","last_unstale":"2023-11-16T23:58:51.416962+0000","last_undegraded":"2023-11-16T23:58:51.416962+0000","last_fullsized":"2023-11-16T23:58:51.416962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:54.017294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733611+0000","last_change":"2023-11-16T23:58:45.715656+0000","last_active":"2023-11-16T23:59:32.733611+0000","last_peered":"2023-11-16T23:59:32.733611+0000","last_clean":"2023-11-16T23:59:32.733611+0000","last_became_active":"2023-11-16T23:58:45.715331+0000","last_became_peered":"2023-11-16T23:58:45.715331+0000","last_unstale":"2023-11-16T23:59:32.733611+0000","last_undegraded":"2023-11-16T23:59:32.733611+0000","last_fullsized":"2023-11-16T23:59:32.733611+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:38:28.590646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973481+0000","last_change":"2023-11-16T23:58:46.294442+0000","last_active":"2023-11-16T23:58:51.973481+0000","last_peered":"2023-11-16T23:58:51.973481+0000","last_clean":"2023-11-16T23:58:51.973481+0000","last_became_active":"2023-11-16T23:58:46.294112+0000","last_became_peered":"2023-11-16T23:58:46.294112+0000","last_unstale":"2023-11-16T23:58:51.973481+0000","last_undegraded":"2023-11-16T23:58:51.973481+0000","last_fullsized":"2023-11-16T23:58:51.973481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:16:37.966411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416678+0000","last_change":"2023-11-16T23:58:50.747104+0000","last_active":"2023-11-16T23:58:51.416678+0000","last_peered":"2023-11-16T23:58:51.416678+0000","last_clean":"2023-11-16T23:58:51.416678+0000","last_became_active":"2023-11-16T23:58:50.746684+0000","last_became_peered":"2023-11-16T23:58:50.746684+0000","last_unstale":"2023-11-16T23:58:51.416678+0000","last_undegraded":"2023-11-16T23:58:51.416678+0000","last_fullsized":"2023-11-16T23:58:51.416678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:25.519731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715891+0000","last_change":"2023-11-16T23:58:45.719225+0000","last_active":"2023-11-16T23:59:33.715891+0000","last_peered":"2023-11-16T23:59:33.715891+0000","last_clean":"2023-11-16T23:59:33.715891+0000","last_became_active":"2023-11-16T23:58:45.718815+0000","last_became_peered":"2023-11-16T23:58:45.718815+0000","last_unstale":"2023-11-16T23:59:33.715891+0000","last_undegraded":"2023-11-16T23:59:33.715891+0000","last_fullsized":"2023-11-16T23:59:33.715891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:09.302556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583967+0000","last_change":"2023-11-16T23:58:46.292334+0000","last_active":"2023-11-16T23:58:51.583967+0000","last_peered":"2023-11-16T23:58:51.583967+0000","last_clean":"2023-11-16T23:58:51.583967+0000","last_became_active":"2023-11-16T23:58:46.291882+0000","last_became_peered":"2023-11-16T23:58:46.291882+0000","last_unstale":"2023-11-16T23:58:51.583967+0000","last_undegraded":"2023-11-16T23:58:51.583967+0000","last_fullsized":"2023-11-16T23:58:51.583967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:17.751014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241151+0000","last_change":"2023-11-16T23:58:50.742737+0000","last_active":"2023-11-16T23:58:51.241151+0000","last_peered":"2023-11-16T23:58:51.241151+0000","last_clean":"2023-11-16T23:58:51.241151+0000","last_became_active":"2023-11-16T23:58:50.742257+0000","last_became_peered":"2023-11-16T23:58:50.742257+0000","last_unstale":"2023-11-16T23:58:51.241151+0000","last_undegraded":"2023-11-16T23:58:51.241151+0000","last_fullsized":"2023-11-16T23:58:51.241151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:47:46.061379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715391+0000","last_change":"2023-11-16T23:58:45.716537+0000","last_active":"2023-11-16T23:59:33.715391+0000","last_peered":"2023-11-16T23:59:33.715391+0000","last_clean":"2023-11-16T23:59:33.715391+0000","last_became_active":"2023-11-16T23:58:45.716018+0000","last_became_peered":"2023-11-16T23:58:45.716018+0000","last_unstale":"2023-11-16T23:59:33.715391+0000","last_undegraded":"2023-11-16T23:59:33.715391+0000","last_fullsized":"2023-11-16T23:59:33.715391+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:56:57.891968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973828+0000","last_change":"2023-11-16T23:58:46.288713+0000","last_active":"2023-11-16T23:58:51.973828+0000","last_peered":"2023-11-16T23:58:51.973828+0000","last_clean":"2023-11-16T23:58:51.973828+0000","last_became_active":"2023-11-16T23:58:46.288363+0000","last_became_peered":"2023-11-16T23:58:46.288363+0000","last_unstale":"2023-11-16T23:58:51.973828+0000","last_undegraded":"2023-11-16T23:58:51.973828+0000","last_fullsized":"2023-11-16T23:58:51.973828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:48:49.725514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243013+0000","last_change":"2023-11-16T23:58:50.728730+0000","last_active":"2023-11-16T23:58:51.243013+0000","last_peered":"2023-11-16T23:58:51.243013+0000","last_clean":"2023-11-16T23:58:51.243013+0000","last_became_active":"2023-11-16T23:58:50.728164+0000","last_became_peered":"2023-11-16T23:58:50.728164+0000","last_unstale":"2023-11-16T23:58:51.243013+0000","last_undegraded":"2023-11-16T23:58:51.243013+0000","last_fullsized":"2023-11-16T23:58:51.243013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:30:15.323131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583464+0000","last_change":"2023-11-16T23:58:45.713056+0000","last_active":"2023-11-16T23:58:51.583464+0000","last_peered":"2023-11-16T23:58:51.583464+0000","last_clean":"2023-11-16T23:58:51.583464+0000","last_became_active":"2023-11-16T23:58:45.712685+0000","last_became_peered":"2023-11-16T23:58:45.712685+0000","last_unstale":"2023-11-16T23:58:51.583464+0000","last_undegraded":"2023-11-16T23:58:51.583464+0000","last_fullsized":"2023-11-16T23:58:51.583464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:35.811313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974076+0000","last_change":"2023-11-16T23:58:46.291540+0000","last_active":"2023-11-16T23:58:51.974076+0000","last_peered":"2023-11-16T23:58:51.974076+0000","last_clean":"2023-11-16T23:58:51.974076+0000","last_became_active":"2023-11-16T23:58:46.289372+0000","last_became_peered":"2023-11-16T23:58:46.289372+0000","last_unstale":"2023-11-16T23:58:51.974076+0000","last_undegraded":"2023-11-16T23:58:51.974076+0000","last_fullsized":"2023-11-16T23:58:51.974076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:38:19.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974448+0000","last_change":"2023-11-16T23:58:50.729362+0000","last_active":"2023-11-16T23:58:51.974448+0000","last_peered":"2023-11-16T23:58:51.974448+0000","last_clean":"2023-11-16T23:58:51.974448+0000","last_became_active":"2023-11-16T23:58:50.729007+0000","last_became_peered":"2023-11-16T23:58:50.729007+0000","last_unstale":"2023-11-16T23:58:51.974448+0000","last_undegraded":"2023-11-16T23:58:51.974448+0000","last_fullsized":"2023-11-16T23:58:51.974448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:59.110409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734157+0000","last_change":"2023-11-16T23:58:45.717814+0000","last_active":"2023-11-16T23:59:32.734157+0000","last_peered":"2023-11-16T23:59:32.734157+0000","last_clean":"2023-11-16T23:59:32.734157+0000","last_became_active":"2023-11-16T23:58:45.717494+0000","last_became_peered":"2023-11-16T23:58:45.717494+0000","last_unstale":"2023-11-16T23:59:32.734157+0000","last_undegraded":"2023-11-16T23:59:32.734157+0000","last_fullsized":"2023-11-16T23:59:32.734157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:33:17.157376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974442+0000","last_change":"2023-11-16T23:58:46.285305+0000","last_active":"2023-11-16T23:58:51.974442+0000","last_peered":"2023-11-16T23:58:51.974442+0000","last_clean":"2023-11-16T23:58:51.974442+0000","last_became_active":"2023-11-16T23:58:46.284563+0000","last_became_peered":"2023-11-16T23:58:46.284563+0000","last_unstale":"2023-11-16T23:58:51.974442+0000","last_undegraded":"2023-11-16T23:58:51.974442+0000","last_fullsized":"2023-11-16T23:58:51.974442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:12.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973969+0000","last_change":"2023-11-16T23:58:50.740863+0000","last_active":"2023-11-16T23:58:51.973969+0000","last_peered":"2023-11-16T23:58:51.973969+0000","last_clean":"2023-11-16T23:58:51.973969+0000","last_became_active":"2023-11-16T23:58:50.740400+0000","last_became_peered":"2023-11-16T23:58:50.740400+0000","last_unstale":"2023-11-16T23:58:51.973969+0000","last_undegraded":"2023-11-16T23:58:51.973969+0000","last_fullsized":"2023-11-16T23:58:51.973969+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:52:44.691356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239474+0000","last_change":"2023-11-16T23:58:45.715701+0000","last_active":"2023-11-16T23:58:51.239474+0000","last_peered":"2023-11-16T23:58:51.239474+0000","last_clean":"2023-11-16T23:58:51.239474+0000","last_became_active":"2023-11-16T23:58:45.715273+0000","last_became_peered":"2023-11-16T23:58:45.715273+0000","last_unstale":"2023-11-16T23:58:51.239474+0000","last_undegraded":"2023-11-16T23:58:51.239474+0000","last_fullsized":"2023-11-16T23:58:51.239474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:08.019018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243105+0000","last_change":"2023-11-16T23:58:46.294740+0000","last_active":"2023-11-16T23:58:51.243105+0000","last_peered":"2023-11-16T23:58:51.243105+0000","last_clean":"2023-11-16T23:58:51.243105+0000","last_became_active":"2023-11-16T23:58:46.282681+0000","last_became_peered":"2023-11-16T23:58:46.282681+0000","last_unstale":"2023-11-16T23:58:51.243105+0000","last_undegraded":"2023-11-16T23:58:51.243105+0000","last_fullsized":"2023-11-16T23:58:51.243105+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:44.418038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975559+0000","last_change":"2023-11-16T23:58:50.730244+0000","last_active":"2023-11-16T23:58:51.975559+0000","last_peered":"2023-11-16T23:58:51.975559+0000","last_clean":"2023-11-16T23:58:51.975559+0000","last_became_active":"2023-11-16T23:58:50.729669+0000","last_became_peered":"2023-11-16T23:58:50.729669+0000","last_unstale":"2023-11-16T23:58:51.975559+0000","last_undegraded":"2023-11-16T23:58:51.975559+0000","last_fullsized":"2023-11-16T23:58:51.975559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:47:57.416754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584137+0000","last_change":"2023-11-16T23:58:45.722939+0000","last_active":"2023-11-16T23:58:51.584137+0000","last_peered":"2023-11-16T23:58:51.584137+0000","last_clean":"2023-11-16T23:58:51.584137+0000","last_became_active":"2023-11-16T23:58:45.722559+0000","last_became_peered":"2023-11-16T23:58:45.722559+0000","last_unstale":"2023-11-16T23:58:51.584137+0000","last_undegraded":"2023-11-16T23:58:51.584137+0000","last_fullsized":"2023-11-16T23:58:51.584137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:38.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974381+0000","last_change":"2023-11-16T23:58:46.285249+0000","last_active":"2023-11-16T23:58:51.974381+0000","last_peered":"2023-11-16T23:58:51.974381+0000","last_clean":"2023-11-16T23:58:51.974381+0000","last_became_active":"2023-11-16T23:58:46.284815+0000","last_became_peered":"2023-11-16T23:58:46.284815+0000","last_unstale":"2023-11-16T23:58:51.974381+0000","last_undegraded":"2023-11-16T23:58:51.974381+0000","last_fullsized":"2023-11-16T23:58:51.974381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:28.991224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974731+0000","last_change":"2023-11-16T23:58:50.734889+0000","last_active":"2023-11-16T23:58:51.974731+0000","last_peered":"2023-11-16T23:58:51.974731+0000","last_clean":"2023-11-16T23:58:51.974731+0000","last_became_active":"2023-11-16T23:58:50.734461+0000","last_became_peered":"2023-11-16T23:58:50.734461+0000","last_unstale":"2023-11-16T23:58:51.974731+0000","last_undegraded":"2023-11-16T23:58:51.974731+0000","last_fullsized":"2023-11-16T23:58:51.974731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:34:05.742581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'12","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.708283+0000","last_change":"2023-11-16T23:58:45.716428+0000","last_active":"2023-11-16T23:59:33.708283+0000","last_peered":"2023-11-16T23:59:33.708283+0000","last_clean":"2023-11-16T23:59:33.708283+0000","last_became_active":"2023-11-16T23:58:45.716076+0000","last_became_peered":"2023-11-16T23:58:45.716076+0000","last_unstale":"2023-11-16T23:59:33.708283+0000","last_undegraded":"2023-11-16T23:59:33.708283+0000","last_fullsized":"2023-11-16T23:59:33.708283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:14:06.840585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973844+0000","last_change":"2023-11-16T23:58:46.282889+0000","last_active":"2023-11-16T23:58:51.973844+0000","last_peered":"2023-11-16T23:58:51.973844+0000","last_clean":"2023-11-16T23:58:51.973844+0000","last_became_active":"2023-11-16T23:58:46.282450+0000","last_became_peered":"2023-11-16T23:58:46.282450+0000","last_unstale":"2023-11-16T23:58:51.973844+0000","last_undegraded":"2023-11-16T23:58:51.973844+0000","last_fullsized":"2023-11-16T23:58:51.973844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:08:09.378619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584460+0000","last_change":"2023-11-16T23:58:50.735438+0000","last_active":"2023-11-16T23:58:51.584460+0000","last_peered":"2023-11-16T23:58:51.584460+0000","last_clean":"2023-11-16T23:58:51.584460+0000","last_became_active":"2023-11-16T23:58:50.735040+0000","last_became_peered":"2023-11-16T23:58:50.735040+0000","last_unstale":"2023-11-16T23:58:51.584460+0000","last_undegraded":"2023-11-16T23:58:51.584460+0000","last_fullsized":"2023-11-16T23:58:51.584460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:20.496063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734379+0000","last_change":"2023-11-16T23:58:45.721161+0000","last_active":"2023-11-16T23:59:32.734379+0000","last_peered":"2023-11-16T23:59:32.734379+0000","last_clean":"2023-11-16T23:59:32.734379+0000","last_became_active":"2023-11-16T23:58:45.720826+0000","last_became_peered":"2023-11-16T23:58:45.720826+0000","last_unstale":"2023-11-16T23:59:32.734379+0000","last_undegraded":"2023-11-16T23:59:32.734379+0000","last_fullsized":"2023-11-16T23:59:32.734379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:33.101318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974461+0000","last_change":"2023-11-16T23:58:46.293152+0000","last_active":"2023-11-16T23:58:51.974461+0000","last_peered":"2023-11-16T23:58:51.974461+0000","last_clean":"2023-11-16T23:58:51.974461+0000","last_became_active":"2023-11-16T23:58:46.292830+0000","last_became_peered":"2023-11-16T23:58:46.292830+0000","last_unstale":"2023-11-16T23:58:51.974461+0000","last_undegraded":"2023-11-16T23:58:51.974461+0000","last_fullsized":"2023-11-16T23:58:51.974461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:06.803488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240824+0000","last_change":"2023-11-16T23:58:50.739327+0000","last_active":"2023-11-16T23:58:51.240824+0000","last_peered":"2023-11-16T23:58:51.240824+0000","last_clean":"2023-11-16T23:58:51.240824+0000","last_became_active":"2023-11-16T23:58:50.739024+0000","last_became_peered":"2023-11-16T23:58:50.739024+0000","last_unstale":"2023-11-16T23:58:51.240824+0000","last_undegraded":"2023-11-16T23:58:51.240824+0000","last_fullsized":"2023-11-16T23:58:51.240824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:58.095409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734178+0000","last_change":"2023-11-16T23:58:45.716730+0000","last_active":"2023-11-16T23:59:32.734178+0000","last_peered":"2023-11-16T23:59:32.734178+0000","last_clean":"2023-11-16T23:59:32.734178+0000","last_became_active":"2023-11-16T23:58:45.716380+0000","last_became_peered":"2023-11-16T23:58:45.716380+0000","last_unstale":"2023-11-16T23:59:32.734178+0000","last_undegraded":"2023-11-16T23:59:32.734178+0000","last_fullsized":"2023-11-16T23:59:32.734178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:06:26.975787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239189+0000","last_change":"2023-11-16T23:58:46.290867+0000","last_active":"2023-11-16T23:58:51.239189+0000","last_peered":"2023-11-16T23:58:51.239189+0000","last_clean":"2023-11-16T23:58:51.239189+0000","last_became_active":"2023-11-16T23:58:46.290464+0000","last_became_peered":"2023-11-16T23:58:46.290464+0000","last_unstale":"2023-11-16T23:58:51.239189+0000","last_undegraded":"2023-11-16T23:58:51.239189+0000","last_fullsized":"2023-11-16T23:58:51.239189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:23.908610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975901+0000","last_change":"2023-11-16T23:58:50.747045+0000","last_active":"2023-11-16T23:58:51.975901+0000","last_peered":"2023-11-16T23:58:51.975901+0000","last_clean":"2023-11-16T23:58:51.975901+0000","last_became_active":"2023-11-16T23:58:50.743955+0000","last_became_peered":"2023-11-16T23:58:50.743955+0000","last_unstale":"2023-11-16T23:58:51.975901+0000","last_undegraded":"2023-11-16T23:58:51.975901+0000","last_fullsized":"2023-11-16T23:58:51.975901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:54:18.403005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:34.720344+0000","last_change":"2023-11-16T23:58:45.714767+0000","last_active":"2023-11-16T23:59:34.720344+0000","last_peered":"2023-11-16T23:59:34.720344+0000","last_clean":"2023-11-16T23:59:34.720344+0000","last_became_active":"2023-11-16T23:58:45.713817+0000","last_became_peered":"2023-11-16T23:58:45.713817+0000","last_unstale":"2023-11-16T23:59:34.720344+0000","last_undegraded":"2023-11-16T23:59:34.720344+0000","last_fullsized":"2023-11-16T23:59:34.720344+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:44:48.694382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238988+0000","last_change":"2023-11-16T23:58:46.295593+0000","last_active":"2023-11-16T23:58:51.238988+0000","last_peered":"2023-11-16T23:58:51.238988+0000","last_clean":"2023-11-16T23:58:51.238988+0000","last_became_active":"2023-11-16T23:58:46.295253+0000","last_became_peered":"2023-11-16T23:58:46.295253+0000","last_unstale":"2023-11-16T23:58:51.238988+0000","last_undegraded":"2023-11-16T23:58:51.238988+0000","last_fullsized":"2023-11-16T23:58:51.238988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:57:11.390297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976616+0000","last_change":"2023-11-16T23:58:50.730370+0000","last_active":"2023-11-16T23:58:51.976616+0000","last_peered":"2023-11-16T23:58:51.976616+0000","last_clean":"2023-11-16T23:58:51.976616+0000","last_became_active":"2023-11-16T23:58:50.729915+0000","last_became_peered":"2023-11-16T23:58:50.729915+0000","last_unstale":"2023-11-16T23:58:51.976616+0000","last_undegraded":"2023-11-16T23:58:51.976616+0000","last_fullsized":"2023-11-16T23:58:51.976616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:40:19.878796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718645+0000","last_change":"2023-11-16T23:58:45.715792+0000","last_active":"2023-11-16T23:59:33.718645+0000","last_peered":"2023-11-16T23:59:33.718645+0000","last_clean":"2023-11-16T23:59:33.718645+0000","last_became_active":"2023-11-16T23:58:45.715454+0000","last_became_peered":"2023-11-16T23:58:45.715454+0000","last_unstale":"2023-11-16T23:59:33.718645+0000","last_undegraded":"2023-11-16T23:59:33.718645+0000","last_fullsized":"2023-11-16T23:59:33.718645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:51.440176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583583+0000","last_change":"2023-11-16T23:58:46.288249+0000","last_active":"2023-11-16T23:58:51.583583+0000","last_peered":"2023-11-16T23:58:51.583583+0000","last_clean":"2023-11-16T23:58:51.583583+0000","last_became_active":"2023-11-16T23:58:46.287916+0000","last_became_peered":"2023-11-16T23:58:46.287916+0000","last_unstale":"2023-11-16T23:58:51.583583+0000","last_undegraded":"2023-11-16T23:58:51.583583+0000","last_fullsized":"2023-11-16T23:58:51.583583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:34:19.800012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583795+0000","last_change":"2023-11-16T23:58:50.733835+0000","last_active":"2023-11-16T23:58:51.583795+0000","last_peered":"2023-11-16T23:58:51.583795+0000","last_clean":"2023-11-16T23:58:51.583795+0000","last_became_active":"2023-11-16T23:58:50.733373+0000","last_became_peered":"2023-11-16T23:58:50.733373+0000","last_unstale":"2023-11-16T23:58:51.583795+0000","last_undegraded":"2023-11-16T23:58:51.583795+0000","last_fullsized":"2023-11-16T23:58:51.583795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:27:17.246864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973964+0000","last_change":"2023-11-16T23:58:45.712055+0000","last_active":"2023-11-16T23:58:51.973964+0000","last_peered":"2023-11-16T23:58:51.973964+0000","last_clean":"2023-11-16T23:58:51.973964+0000","last_became_active":"2023-11-16T23:58:45.711618+0000","last_became_peered":"2023-11-16T23:58:45.711618+0000","last_unstale":"2023-11-16T23:58:51.973964+0000","last_undegraded":"2023-11-16T23:58:51.973964+0000","last_fullsized":"2023-11-16T23:58:51.973964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:37:41.700022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241050+0000","last_change":"2023-11-16T23:58:46.291460+0000","last_active":"2023-11-16T23:58:51.241050+0000","last_peered":"2023-11-16T23:58:51.241050+0000","last_clean":"2023-11-16T23:58:51.241050+0000","last_became_active":"2023-11-16T23:58:46.291144+0000","last_became_peered":"2023-11-16T23:58:46.291144+0000","last_unstale":"2023-11-16T23:58:51.241050+0000","last_undegraded":"2023-11-16T23:58:51.241050+0000","last_fullsized":"2023-11-16T23:58:51.241050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:24:10.888038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973559+0000","last_change":"2023-11-16T23:58:50.927320+0000","last_active":"2023-11-16T23:58:51.973559+0000","last_peered":"2023-11-16T23:58:51.973559+0000","last_clean":"2023-11-16T23:58:51.973559+0000","last_became_active":"2023-11-16T23:58:50.926767+0000","last_became_peered":"2023-11-16T23:58:50.926767+0000","last_unstale":"2023-11-16T23:58:51.973559+0000","last_undegraded":"2023-11-16T23:58:51.973559+0000","last_fullsized":"2023-11-16T23:58:51.973559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:21.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975281+0000","last_change":"2023-11-16T23:58:45.715654+0000","last_active":"2023-11-16T23:58:51.975281+0000","last_peered":"2023-11-16T23:58:51.975281+0000","last_clean":"2023-11-16T23:58:51.975281+0000","last_became_active":"2023-11-16T23:58:45.714899+0000","last_became_peered":"2023-11-16T23:58:45.714899+0000","last_unstale":"2023-11-16T23:58:51.975281+0000","last_undegraded":"2023-11-16T23:58:51.975281+0000","last_fullsized":"2023-11-16T23:58:51.975281+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:53.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974886+0000","last_change":"2023-11-16T23:58:46.291818+0000","last_active":"2023-11-16T23:58:51.974886+0000","last_peered":"2023-11-16T23:58:51.974886+0000","last_clean":"2023-11-16T23:58:51.974886+0000","last_became_active":"2023-11-16T23:58:46.291517+0000","last_became_peered":"2023-11-16T23:58:46.291517+0000","last_unstale":"2023-11-16T23:58:51.974886+0000","last_undegraded":"2023-11-16T23:58:51.974886+0000","last_fullsized":"2023-11-16T23:58:51.974886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:51:34.124036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:50.733546+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:50.733009+0000","last_became_peered":"2023-11-16T23:58:50.733009+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:37:16.925270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239181+0000","last_change":"2023-11-16T23:58:45.719181+0000","last_active":"2023-11-16T23:58:51.239181+0000","last_peered":"2023-11-16T23:58:51.239181+0000","last_clean":"2023-11-16T23:58:51.239181+0000","last_became_active":"2023-11-16T23:58:45.718826+0000","last_became_peered":"2023-11-16T23:58:45.718826+0000","last_unstale":"2023-11-16T23:58:51.239181+0000","last_undegraded":"2023-11-16T23:58:51.239181+0000","last_fullsized":"2023-11-16T23:58:51.239181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:52:25.180163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975356+0000","last_change":"2023-11-16T23:58:46.283405+0000","last_active":"2023-11-16T23:58:51.975356+0000","last_peered":"2023-11-16T23:58:51.975356+0000","last_clean":"2023-11-16T23:58:51.975356+0000","last_became_active":"2023-11-16T23:58:46.282995+0000","last_became_peered":"2023-11-16T23:58:46.282995+0000","last_unstale":"2023-11-16T23:58:51.975356+0000","last_undegraded":"2023-11-16T23:58:51.975356+0000","last_fullsized":"2023-11-16T23:58:51.975356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:53:28.645169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976551+0000","last_change":"2023-11-16T23:58:50.928220+0000","last_active":"2023-11-16T23:58:51.976551+0000","last_peered":"2023-11-16T23:58:51.976551+0000","last_clean":"2023-11-16T23:58:51.976551+0000","last_became_active":"2023-11-16T23:58:50.927006+0000","last_became_peered":"2023-11-16T23:58:50.927006+0000","last_unstale":"2023-11-16T23:58:51.976551+0000","last_undegraded":"2023-11-16T23:58:51.976551+0000","last_fullsized":"2023-11-16T23:58:51.976551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:09:21.135371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583555+0000","last_change":"2023-11-16T23:58:46.289260+0000","last_active":"2023-11-16T23:58:51.583555+0000","last_peered":"2023-11-16T23:58:51.583555+0000","last_clean":"2023-11-16T23:58:51.583555+0000","last_became_active":"2023-11-16T23:58:46.285041+0000","last_became_peered":"2023-11-16T23:58:46.285041+0000","last_unstale":"2023-11-16T23:58:51.583555+0000","last_undegraded":"2023-11-16T23:58:51.583555+0000","last_fullsized":"2023-11-16T23:58:51.583555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:16.675991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718821+0000","last_change":"2023-11-16T23:58:45.726423+0000","last_active":"2023-11-16T23:59:33.718821+0000","last_peered":"2023-11-16T23:59:33.718821+0000","last_clean":"2023-11-16T23:59:33.718821+0000","last_became_active":"2023-11-16T23:58:45.725996+0000","last_became_peered":"2023-11-16T23:58:45.725996+0000","last_unstale":"2023-11-16T23:59:33.718821+0000","last_undegraded":"2023-11-16T23:59:33.718821+0000","last_fullsized":"2023-11-16T23:59:33.718821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:38.058418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239536+0000","last_change":"2023-11-16T23:58:50.730148+0000","last_active":"2023-11-16T23:58:51.239536+0000","last_peered":"2023-11-16T23:58:51.239536+0000","last_clean":"2023-11-16T23:58:51.239536+0000","last_became_active":"2023-11-16T23:58:50.729675+0000","last_became_peered":"2023-11-16T23:58:50.729675+0000","last_unstale":"2023-11-16T23:58:51.239536+0000","last_undegraded":"2023-11-16T23:58:51.239536+0000","last_fullsized":"2023-11-16T23:58:51.239536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:42:47.288350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974143+0000","last_change":"2023-11-16T23:58:46.295437+0000","last_active":"2023-11-16T23:58:51.974143+0000","last_peered":"2023-11-16T23:58:51.974143+0000","last_clean":"2023-11-16T23:58:51.974143+0000","last_became_active":"2023-11-16T23:58:46.291775+0000","last_became_peered":"2023-11-16T23:58:46.291775+0000","last_unstale":"2023-11-16T23:58:51.974143+0000","last_undegraded":"2023-11-16T23:58:51.974143+0000","last_fullsized":"2023-11-16T23:58:51.974143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:17.517488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714142+0000","last_change":"2023-11-16T23:58:45.714497+0000","last_active":"2023-11-16T23:59:33.714142+0000","last_peered":"2023-11-16T23:59:33.714142+0000","last_clean":"2023-11-16T23:59:33.714142+0000","last_became_active":"2023-11-16T23:58:45.714167+0000","last_became_peered":"2023-11-16T23:58:45.714167+0000","last_unstale":"2023-11-16T23:59:33.714142+0000","last_undegraded":"2023-11-16T23:59:33.714142+0000","last_fullsized":"2023-11-16T23:59:33.714142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:46.839077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238802+0000","last_change":"2023-11-16T23:57:37.700941+0000","last_active":"2023-11-16T23:58:51.238802+0000","last_peered":"2023-11-16T23:58:51.238802+0000","last_clean":"2023-11-16T23:58:51.238802+0000","last_became_active":"2023-11-16T23:57:37.700379+0000","last_became_peered":"2023-11-16T23:57:37.700379+0000","last_unstale":"2023-11-16T23:58:51.238802+0000","last_undegraded":"2023-11-16T23:58:51.238802+0000","last_fullsized":"2023-11-16T23:58:51.238802+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_clean_scrub_stamp":"2023-11-16T23:54:03.597877+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:51:56.505655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584217+0000","last_change":"2023-11-16T23:58:50.927446+0000","last_active":"2023-11-16T23:58:51.584217+0000","last_peered":"2023-11-16T23:58:51.584217+0000","last_clean":"2023-11-16T23:58:51.584217+0000","last_became_active":"2023-11-16T23:58:50.927042+0000","last_became_peered":"2023-11-16T23:58:50.927042+0000","last_unstale":"2023-11-16T23:58:51.584217+0000","last_undegraded":"2023-11-16T23:58:51.584217+0000","last_fullsized":"2023-11-16T23:58:51.584217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:11:06.468189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973431+0000","last_change":"2023-11-16T23:58:46.283611+0000","last_active":"2023-11-16T23:58:51.973431+0000","last_peered":"2023-11-16T23:58:51.973431+0000","last_clean":"2023-11-16T23:58:51.973431+0000","last_became_active":"2023-11-16T23:58:46.281685+0000","last_became_peered":"2023-11-16T23:58:46.281685+0000","last_unstale":"2023-11-16T23:58:51.973431+0000","last_undegraded":"2023-11-16T23:58:51.973431+0000","last_fullsized":"2023-11-16T23:58:51.973431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:28:30.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238863+0000","last_change":"2023-11-16T23:58:45.716094+0000","last_active":"2023-11-16T23:58:51.238863+0000","last_peered":"2023-11-16T23:58:51.238863+0000","last_clean":"2023-11-16T23:58:51.238863+0000","last_became_active":"2023-11-16T23:58:45.715748+0000","last_became_peered":"2023-11-16T23:58:45.715748+0000","last_unstale":"2023-11-16T23:58:51.238863+0000","last_undegraded":"2023-11-16T23:58:51.238863+0000","last_fullsized":"2023-11-16T23:58:51.238863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:48.184073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240803+0000","last_change":"2023-11-16T23:58:50.737281+0000","last_active":"2023-11-16T23:58:51.240803+0000","last_peered":"2023-11-16T23:58:51.240803+0000","last_clean":"2023-11-16T23:58:51.240803+0000","last_became_active":"2023-11-16T23:58:50.736797+0000","last_became_peered":"2023-11-16T23:58:50.736797+0000","last_unstale":"2023-11-16T23:58:51.240803+0000","last_undegraded":"2023-11-16T23:58:51.240803+0000","last_fullsized":"2023-11-16T23:58:51.240803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:39.192922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734179+0000","last_change":"2023-11-16T23:58:45.722935+0000","last_active":"2023-11-16T23:59:32.734179+0000","last_peered":"2023-11-16T23:59:32.734179+0000","last_clean":"2023-11-16T23:59:32.734179+0000","last_became_active":"2023-11-16T23:58:45.722489+0000","last_became_peered":"2023-11-16T23:58:45.722489+0000","last_unstale":"2023-11-16T23:59:32.734179+0000","last_undegraded":"2023-11-16T23:59:32.734179+0000","last_fullsized":"2023-11-16T23:59:32.734179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:58.547593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975877+0000","last_change":"2023-11-16T23:58:46.284611+0000","last_active":"2023-11-16T23:58:51.975877+0000","last_peered":"2023-11-16T23:58:51.975877+0000","last_clean":"2023-11-16T23:58:51.975877+0000","last_became_active":"2023-11-16T23:58:46.283063+0000","last_became_peered":"2023-11-16T23:58:46.283063+0000","last_unstale":"2023-11-16T23:58:51.975877+0000","last_undegraded":"2023-11-16T23:58:51.975877+0000","last_fullsized":"2023-11-16T23:58:51.975877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:48:34.246739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973742+0000","last_change":"2023-11-16T23:58:50.927654+0000","last_active":"2023-11-16T23:58:51.973742+0000","last_peered":"2023-11-16T23:58:51.973742+0000","last_clean":"2023-11-16T23:58:51.973742+0000","last_became_active":"2023-11-16T23:58:50.926936+0000","last_became_peered":"2023-11-16T23:58:50.926936+0000","last_unstale":"2023-11-16T23:58:51.973742+0000","last_undegraded":"2023-11-16T23:58:51.973742+0000","last_fullsized":"2023-11-16T23:58:51.973742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:20.080380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717677+0000","last_change":"2023-11-16T23:58:45.717981+0000","last_active":"2023-11-16T23:59:33.717677+0000","last_peered":"2023-11-16T23:59:33.717677+0000","last_clean":"2023-11-16T23:59:33.717677+0000","last_became_active":"2023-11-16T23:58:45.717702+0000","last_became_peered":"2023-11-16T23:58:45.717702+0000","last_unstale":"2023-11-16T23:59:33.717677+0000","last_undegraded":"2023-11-16T23:59:33.717677+0000","last_fullsized":"2023-11-16T23:59:33.717677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:32:06.566478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974772+0000","last_change":"2023-11-16T23:58:46.288768+0000","last_active":"2023-11-16T23:58:51.974772+0000","last_peered":"2023-11-16T23:58:51.974772+0000","last_clean":"2023-11-16T23:58:51.974772+0000","last_became_active":"2023-11-16T23:58:46.288430+0000","last_became_peered":"2023-11-16T23:58:46.288430+0000","last_unstale":"2023-11-16T23:58:51.974772+0000","last_undegraded":"2023-11-16T23:58:51.974772+0000","last_fullsized":"2023-11-16T23:58:51.974772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:19:06.093201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416385+0000","last_change":"2023-11-16T23:58:50.724748+0000","last_active":"2023-11-16T23:58:51.416385+0000","last_peered":"2023-11-16T23:58:51.416385+0000","last_clean":"2023-11-16T23:58:51.416385+0000","last_became_active":"2023-11-16T23:58:50.724313+0000","last_became_peered":"2023-11-16T23:58:50.724313+0000","last_unstale":"2023-11-16T23:58:51.416385+0000","last_undegraded":"2023-11-16T23:58:51.416385+0000","last_fullsized":"2023-11-16T23:58:51.416385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:17:21.820364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732623+0000","last_change":"2023-11-16T23:58:45.715563+0000","last_active":"2023-11-16T23:59:32.732623+0000","last_peered":"2023-11-16T23:59:32.732623+0000","last_clean":"2023-11-16T23:59:32.732623+0000","last_became_active":"2023-11-16T23:58:45.715217+0000","last_became_peered":"2023-11-16T23:58:45.715217+0000","last_unstale":"2023-11-16T23:59:32.732623+0000","last_undegraded":"2023-11-16T23:59:32.732623+0000","last_fullsized":"2023-11-16T23:59:32.732623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:41.330842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242993+0000","last_change":"2023-11-16T23:58:46.298305+0000","last_active":"2023-11-16T23:58:51.242993+0000","last_peered":"2023-11-16T23:58:51.242993+0000","last_clean":"2023-11-16T23:58:51.242993+0000","last_became_active":"2023-11-16T23:58:46.297444+0000","last_became_peered":"2023-11-16T23:58:46.297444+0000","last_unstale":"2023-11-16T23:58:51.242993+0000","last_undegraded":"2023-11-16T23:58:51.242993+0000","last_fullsized":"2023-11-16T23:58:51.242993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:36.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974077+0000","last_change":"2023-11-16T23:58:50.927915+0000","last_active":"2023-11-16T23:58:51.974077+0000","last_peered":"2023-11-16T23:58:51.974077+0000","last_clean":"2023-11-16T23:58:51.974077+0000","last_became_active":"2023-11-16T23:58:50.927451+0000","last_became_peered":"2023-11-16T23:58:50.927451+0000","last_unstale":"2023-11-16T23:58:51.974077+0000","last_undegraded":"2023-11-16T23:58:51.974077+0000","last_fullsized":"2023-11-16T23:58:51.974077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:21:21.865022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732946+0000","last_change":"2023-11-16T23:58:45.710870+0000","last_active":"2023-11-16T23:59:32.732946+0000","last_peered":"2023-11-16T23:59:32.732946+0000","last_clean":"2023-11-16T23:59:32.732946+0000","last_became_active":"2023-11-16T23:58:45.710536+0000","last_became_peered":"2023-11-16T23:58:45.710536+0000","last_unstale":"2023-11-16T23:59:32.732946+0000","last_undegraded":"2023-11-16T23:59:32.732946+0000","last_fullsized":"2023-11-16T23:59:32.732946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:32.891477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973290+0000","last_change":"2023-11-16T23:58:46.282277+0000","last_active":"2023-11-16T23:58:51.973290+0000","last_peered":"2023-11-16T23:58:51.973290+0000","last_clean":"2023-11-16T23:58:51.973290+0000","last_became_active":"2023-11-16T23:58:46.281913+0000","last_became_peered":"2023-11-16T23:58:46.281913+0000","last_unstale":"2023-11-16T23:58:51.973290+0000","last_undegraded":"2023-11-16T23:58:51.973290+0000","last_fullsized":"2023-11-16T23:58:51.973290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:19:12.760160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976155+0000","last_change":"2023-11-16T23:58:50.740914+0000","last_active":"2023-11-16T23:58:51.976155+0000","last_peered":"2023-11-16T23:58:51.976155+0000","last_clean":"2023-11-16T23:58:51.976155+0000","last_became_active":"2023-11-16T23:58:50.740348+0000","last_became_peered":"2023-11-16T23:58:50.740348+0000","last_unstale":"2023-11-16T23:58:51.976155+0000","last_undegraded":"2023-11-16T23:58:51.976155+0000","last_fullsized":"2023-11-16T23:58:51.976155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:59:30.440820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734107+0000","last_change":"2023-11-16T23:58:45.723953+0000","last_active":"2023-11-16T23:59:32.734107+0000","last_peered":"2023-11-16T23:59:32.734107+0000","last_clean":"2023-11-16T23:59:32.734107+0000","last_became_active":"2023-11-16T23:58:45.722075+0000","last_became_peered":"2023-11-16T23:58:45.722075+0000","last_unstale":"2023-11-16T23:59:32.734107+0000","last_undegraded":"2023-11-16T23:59:32.734107+0000","last_fullsized":"2023-11-16T23:59:32.734107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:46:29.950815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240548+0000","last_change":"2023-11-16T23:58:46.290530+0000","last_active":"2023-11-16T23:58:51.240548+0000","last_peered":"2023-11-16T23:58:51.240548+0000","last_clean":"2023-11-16T23:58:51.240548+0000","last_became_active":"2023-11-16T23:58:46.290181+0000","last_became_peered":"2023-11-16T23:58:46.290181+0000","last_unstale":"2023-11-16T23:58:51.240548+0000","last_undegraded":"2023-11-16T23:58:51.240548+0000","last_fullsized":"2023-11-16T23:58:51.240548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:20.504586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974154+0000","last_change":"2023-11-16T23:58:50.739559+0000","last_active":"2023-11-16T23:58:51.974154+0000","last_peered":"2023-11-16T23:58:51.974154+0000","last_clean":"2023-11-16T23:58:51.974154+0000","last_became_active":"2023-11-16T23:58:50.739156+0000","last_became_peered":"2023-11-16T23:58:50.739156+0000","last_unstale":"2023-11-16T23:58:51.974154+0000","last_undegraded":"2023-11-16T23:58:51.974154+0000","last_fullsized":"2023-11-16T23:58:51.974154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:45.312325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416423+0000","last_change":"2023-11-16T23:58:45.723679+0000","last_active":"2023-11-16T23:58:51.416423+0000","last_peered":"2023-11-16T23:58:51.416423+0000","last_clean":"2023-11-16T23:58:51.416423+0000","last_became_active":"2023-11-16T23:58:45.723340+0000","last_became_peered":"2023-11-16T23:58:45.723340+0000","last_unstale":"2023-11-16T23:58:51.416423+0000","last_undegraded":"2023-11-16T23:58:51.416423+0000","last_fullsized":"2023-11-16T23:58:51.416423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:28:17.612030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974645+0000","last_change":"2023-11-16T23:58:46.297004+0000","last_active":"2023-11-16T23:58:51.974645+0000","last_peered":"2023-11-16T23:58:51.974645+0000","last_clean":"2023-11-16T23:58:51.974645+0000","last_became_active":"2023-11-16T23:58:46.296622+0000","last_became_peered":"2023-11-16T23:58:46.296622+0000","last_unstale":"2023-11-16T23:58:51.974645+0000","last_undegraded":"2023-11-16T23:58:51.974645+0000","last_fullsized":"2023-11-16T23:58:51.974645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:00:00.033560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974484+0000","last_change":"2023-11-16T23:58:50.927313+0000","last_active":"2023-11-16T23:58:51.974484+0000","last_peered":"2023-11-16T23:58:51.974484+0000","last_clean":"2023-11-16T23:58:51.974484+0000","last_became_active":"2023-11-16T23:58:50.926865+0000","last_became_peered":"2023-11-16T23:58:50.926865+0000","last_unstale":"2023-11-16T23:58:51.974484+0000","last_undegraded":"2023-11-16T23:58:51.974484+0000","last_fullsized":"2023-11-16T23:58:51.974484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:41:57.718515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733211+0000","last_change":"2023-11-16T23:58:45.714188+0000","last_active":"2023-11-16T23:59:32.733211+0000","last_peered":"2023-11-16T23:59:32.733211+0000","last_clean":"2023-11-16T23:59:32.733211+0000","last_became_active":"2023-11-16T23:58:45.713785+0000","last_became_peered":"2023-11-16T23:58:45.713785+0000","last_unstale":"2023-11-16T23:59:32.733211+0000","last_undegraded":"2023-11-16T23:59:32.733211+0000","last_fullsized":"2023-11-16T23:59:32.733211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:49:36.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239324+0000","last_change":"2023-11-16T23:58:46.291307+0000","last_active":"2023-11-16T23:58:51.239324+0000","last_peered":"2023-11-16T23:58:51.239324+0000","last_clean":"2023-11-16T23:58:51.239324+0000","last_became_active":"2023-11-16T23:58:46.289946+0000","last_became_peered":"2023-11-16T23:58:46.289946+0000","last_unstale":"2023-11-16T23:58:51.239324+0000","last_undegraded":"2023-11-16T23:58:51.239324+0000","last_fullsized":"2023-11-16T23:58:51.239324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:58:49.392838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973947+0000","last_change":"2023-11-16T23:58:50.737205+0000","last_active":"2023-11-16T23:58:51.973947+0000","last_peered":"2023-11-16T23:58:51.973947+0000","last_clean":"2023-11-16T23:58:51.973947+0000","last_became_active":"2023-11-16T23:58:50.736756+0000","last_became_peered":"2023-11-16T23:58:50.736756+0000","last_unstale":"2023-11-16T23:58:51.973947+0000","last_undegraded":"2023-11-16T23:58:51.973947+0000","last_fullsized":"2023-11-16T23:58:51.973947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:05:08.988766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416782+0000","last_change":"2023-11-16T23:58:45.713979+0000","last_active":"2023-11-16T23:58:51.416782+0000","last_peered":"2023-11-16T23:58:51.416782+0000","last_clean":"2023-11-16T23:58:51.416782+0000","last_became_active":"2023-11-16T23:58:45.713492+0000","last_became_peered":"2023-11-16T23:58:45.713492+0000","last_unstale":"2023-11-16T23:58:51.416782+0000","last_undegraded":"2023-11-16T23:58:51.416782+0000","last_fullsized":"2023-11-16T23:58:51.416782+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:17:15.481986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239735+0000","last_change":"2023-11-16T23:58:46.289100+0000","last_active":"2023-11-16T23:58:51.239735+0000","last_peered":"2023-11-16T23:58:51.239735+0000","last_clean":"2023-11-16T23:58:51.239735+0000","last_became_active":"2023-11-16T23:58:46.288506+0000","last_became_peered":"2023-11-16T23:58:46.288506+0000","last_unstale":"2023-11-16T23:58:51.239735+0000","last_undegraded":"2023-11-16T23:58:51.239735+0000","last_fullsized":"2023-11-16T23:58:51.239735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:44:12.486352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242604+0000","last_change":"2023-11-16T23:58:50.741205+0000","last_active":"2023-11-16T23:58:51.242604+0000","last_peered":"2023-11-16T23:58:51.242604+0000","last_clean":"2023-11-16T23:58:51.242604+0000","last_became_active":"2023-11-16T23:58:50.740871+0000","last_became_peered":"2023-11-16T23:58:50.740871+0000","last_unstale":"2023-11-16T23:58:51.242604+0000","last_undegraded":"2023-11-16T23:58:51.242604+0000","last_fullsized":"2023-11-16T23:58:51.242604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:25:17.203133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584250+0000","last_change":"2023-11-16T23:58:45.723618+0000","last_active":"2023-11-16T23:58:51.584250+0000","last_peered":"2023-11-16T23:58:51.584250+0000","last_clean":"2023-11-16T23:58:51.584250+0000","last_became_active":"2023-11-16T23:58:45.723266+0000","last_became_peered":"2023-11-16T23:58:45.723266+0000","last_unstale":"2023-11-16T23:58:51.584250+0000","last_undegraded":"2023-11-16T23:58:51.584250+0000","last_fullsized":"2023-11-16T23:58:51.584250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:20:44.206104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239349+0000","last_change":"2023-11-16T23:58:46.295295+0000","last_active":"2023-11-16T23:58:51.239349+0000","last_peered":"2023-11-16T23:58:51.239349+0000","last_clean":"2023-11-16T23:58:51.239349+0000","last_became_active":"2023-11-16T23:58:46.294801+0000","last_became_peered":"2023-11-16T23:58:46.294801+0000","last_unstale":"2023-11-16T23:58:51.239349+0000","last_undegraded":"2023-11-16T23:58:51.239349+0000","last_fullsized":"2023-11-16T23:58:51.239349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:00.523212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583352+0000","last_change":"2023-11-16T23:58:50.725856+0000","last_active":"2023-11-16T23:58:51.583352+0000","last_peered":"2023-11-16T23:58:51.583352+0000","last_clean":"2023-11-16T23:58:51.583352+0000","last_became_active":"2023-11-16T23:58:50.725453+0000","last_became_peered":"2023-11-16T23:58:50.725453+0000","last_unstale":"2023-11-16T23:58:51.583352+0000","last_undegraded":"2023-11-16T23:58:51.583352+0000","last_fullsized":"2023-11-16T23:58:51.583352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:32:17.547808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976506+0000","last_change":"2023-11-16T23:58:45.718226+0000","last_active":"2023-11-16T23:58:51.976506+0000","last_peered":"2023-11-16T23:58:51.976506+0000","last_clean":"2023-11-16T23:58:51.976506+0000","last_became_active":"2023-11-16T23:58:45.717893+0000","last_became_peered":"2023-11-16T23:58:45.717893+0000","last_unstale":"2023-11-16T23:58:51.976506+0000","last_undegraded":"2023-11-16T23:58:51.976506+0000","last_fullsized":"2023-11-16T23:58:51.976506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:01.114364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239740+0000","last_change":"2023-11-16T23:58:46.289906+0000","last_active":"2023-11-16T23:58:51.239740+0000","last_peered":"2023-11-16T23:58:51.239740+0000","last_clean":"2023-11-16T23:58:51.239740+0000","last_became_active":"2023-11-16T23:58:46.289266+0000","last_became_peered":"2023-11-16T23:58:46.289266+0000","last_unstale":"2023-11-16T23:58:51.239740+0000","last_undegraded":"2023-11-16T23:58:51.239740+0000","last_fullsized":"2023-11-16T23:58:51.239740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:54:03.586113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240620+0000","last_change":"2023-11-16T23:58:46.292220+0000","last_active":"2023-11-16T23:58:51.240620+0000","last_peered":"2023-11-16T23:58:51.240620+0000","last_clean":"2023-11-16T23:58:51.240620+0000","last_became_active":"2023-11-16T23:58:46.291840+0000","last_became_peered":"2023-11-16T23:58:46.291840+0000","last_unstale":"2023-11-16T23:58:51.240620+0000","last_undegraded":"2023-11-16T23:58:51.240620+0000","last_fullsized":"2023-11-16T23:58:51.240620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:18:35.280653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416702+0000","last_change":"2023-11-16T23:58:46.287601+0000","last_active":"2023-11-16T23:58:51.416702+0000","last_peered":"2023-11-16T23:58:51.416702+0000","last_clean":"2023-11-16T23:58:51.416702+0000","last_became_active":"2023-11-16T23:58:46.284738+0000","last_became_peered":"2023-11-16T23:58:46.284738+0000","last_unstale":"2023-11-16T23:58:51.416702+0000","last_undegraded":"2023-11-16T23:58:51.416702+0000","last_fullsized":"2023-11-16T23:58:51.416702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:49.913458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416384+0000","last_change":"2023-11-16T23:58:46.288601+0000","last_active":"2023-11-16T23:58:51.416384+0000","last_peered":"2023-11-16T23:58:51.416384+0000","last_clean":"2023-11-16T23:58:51.416384+0000","last_became_active":"2023-11-16T23:58:46.288266+0000","last_became_peered":"2023-11-16T23:58:46.288266+0000","last_unstale":"2023-11-16T23:58:51.416384+0000","last_undegraded":"2023-11-16T23:58:51.416384+0000","last_fullsized":"2023-11-16T23:58:51.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:18.659483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242778+0000","last_change":"2023-11-16T23:58:46.286495+0000","last_active":"2023-11-16T23:58:51.242778+0000","last_peered":"2023-11-16T23:58:51.242778+0000","last_clean":"2023-11-16T23:58:51.242778+0000","last_became_active":"2023-11-16T23:58:46.281157+0000","last_became_peered":"2023-11-16T23:58:46.281157+0000","last_unstale":"2023-11-16T23:58:51.242778+0000","last_undegraded":"2023-11-16T23:58:51.242778+0000","last_fullsized":"2023-11-16T23:58:51.242778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:02.565515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976001+0000","last_change":"2023-11-16T23:58:46.282373+0000","last_active":"2023-11-16T23:58:51.976001+0000","last_peered":"2023-11-16T23:58:51.976001+0000","last_clean":"2023-11-16T23:58:51.976001+0000","last_became_active":"2023-11-16T23:58:46.282028+0000","last_became_peered":"2023-11-16T23:58:46.282028+0000","last_unstale":"2023-11-16T23:58:51.976001+0000","last_undegraded":"2023-11-16T23:58:51.976001+0000","last_fullsized":"2023-11-16T23:58:51.976001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:29:48.640544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241073+0000","last_change":"2023-11-16T23:58:46.285528+0000","last_active":"2023-11-16T23:58:51.241073+0000","last_peered":"2023-11-16T23:58:51.241073+0000","last_clean":"2023-11-16T23:58:51.241073+0000","last_became_active":"2023-11-16T23:58:46.285227+0000","last_became_peered":"2023-11-16T23:58:46.285227+0000","last_unstale":"2023-11-16T23:58:51.241073+0000","last_undegraded":"2023-11-16T23:58:51.241073+0000","last_fullsized":"2023-11-16T23:58:51.241073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:45:11.259431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416982+0000","last_change":"2023-11-16T23:58:46.283514+0000","last_active":"2023-11-16T23:58:51.416982+0000","last_peered":"2023-11-16T23:58:51.416982+0000","last_clean":"2023-11-16T23:58:51.416982+0000","last_became_active":"2023-11-16T23:58:46.283026+0000","last_became_peered":"2023-11-16T23:58:46.283026+0000","last_unstale":"2023-11-16T23:58:51.416982+0000","last_undegraded":"2023-11-16T23:58:51.416982+0000","last_fullsized":"2023-11-16T23:58:51.416982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:14.255094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974891+0000","last_change":"2023-11-16T23:58:46.283334+0000","last_active":"2023-11-16T23:58:51.974891+0000","last_peered":"2023-11-16T23:58:51.974891+0000","last_clean":"2023-11-16T23:58:51.974891+0000","last_became_active":"2023-11-16T23:58:46.283012+0000","last_became_peered":"2023-11-16T23:58:46.283012+0000","last_unstale":"2023-11-16T23:58:51.974891+0000","last_undegraded":"2023-11-16T23:58:51.974891+0000","last_fullsized":"2023-11-16T23:58:51.974891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:44.161087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583457+0000","last_change":"2023-11-16T23:58:46.292720+0000","last_active":"2023-11-16T23:58:51.583457+0000","last_peered":"2023-11-16T23:58:51.583457+0000","last_clean":"2023-11-16T23:58:51.583457+0000","last_became_active":"2023-11-16T23:58:46.292109+0000","last_became_peered":"2023-11-16T23:58:46.292109+0000","last_unstale":"2023-11-16T23:58:51.583457+0000","last_undegraded":"2023-11-16T23:58:51.583457+0000","last_fullsized":"2023-11-16T23:58:51.583457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:41:12.067371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974297+0000","last_change":"2023-11-16T23:58:46.279919+0000","last_active":"2023-11-16T23:58:51.974297+0000","last_peered":"2023-11-16T23:58:51.974297+0000","last_clean":"2023-11-16T23:58:51.974297+0000","last_became_active":"2023-11-16T23:58:46.279601+0000","last_became_peered":"2023-11-16T23:58:46.279601+0000","last_unstale":"2023-11-16T23:58:51.974297+0000","last_undegraded":"2023-11-16T23:58:51.974297+0000","last_fullsized":"2023-11-16T23:58:51.974297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:28:22.080722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975063+0000","last_change":"2023-11-16T23:58:46.293091+0000","last_active":"2023-11-16T23:58:51.975063+0000","last_peered":"2023-11-16T23:58:51.975063+0000","last_clean":"2023-11-16T23:58:51.975063+0000","last_became_active":"2023-11-16T23:58:46.292728+0000","last_became_peered":"2023-11-16T23:58:46.292728+0000","last_unstale":"2023-11-16T23:58:51.975063+0000","last_undegraded":"2023-11-16T23:58:51.975063+0000","last_fullsized":"2023-11-16T23:58:51.975063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:57:24.095879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975716+0000","last_change":"2023-11-16T23:58:46.281708+0000","last_active":"2023-11-16T23:58:51.975716+0000","last_peered":"2023-11-16T23:58:51.975716+0000","last_clean":"2023-11-16T23:58:51.975716+0000","last_became_active":"2023-11-16T23:58:46.281335+0000","last_became_peered":"2023-11-16T23:58:46.281335+0000","last_unstale":"2023-11-16T23:58:51.975716+0000","last_undegraded":"2023-11-16T23:58:51.975716+0000","last_fullsized":"2023-11-16T23:58:51.975716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:46:24.208579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240805+0000","last_change":"2023-11-16T23:58:46.286046+0000","last_active":"2023-11-16T23:58:51.240805+0000","last_peered":"2023-11-16T23:58:51.240805+0000","last_clean":"2023-11-16T23:58:51.240805+0000","last_became_active":"2023-11-16T23:58:46.285702+0000","last_became_peered":"2023-11-16T23:58:46.285702+0000","last_unstale":"2023-11-16T23:58:51.240805+0000","last_undegraded":"2023-11-16T23:58:51.240805+0000","last_fullsized":"2023-11-16T23:58:51.240805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:15:25.406768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":116,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":54053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":79,"seq":339302416405,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":974848,"data_stored":475644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85599999999999998}]},{"osd":7,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.96099999999999997}]},{"osd":10,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.81999999999999995}]}]},{"osd":10,"up_from":71,"seq":304942678041,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738212,"statfs":{"total":95995035648,"available":95987929088,"internally_reserved":0,"allocated":1593344,"data_stored":1065753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.31}]},{"osd":2,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.39600000000000002}]},{"osd":5,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.434},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67800000000000005}]},{"osd":9,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.64700000000000002}]},{"osd":11,"last update":"Thu Nov 16 23:58:58 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.63400000000000001}]}]},{"osd":9,"up_from":66,"seq":283467841570,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6292,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738860,"statfs":{"total":95995035648,"available":95988592640,"internally_reserved":0,"allocated":995328,"data_stored":479976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.434}]},{"osd":1,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.79600000000000004,"15min":0.81200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0669999999999999,"5min":1.139,"15min":1.139},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.97899999999999998,"5min":1.044,"15min":1.044},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.83799999999999997,"15min":0.85599999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.59899999999999998}]},{"osd":4,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.85199999999999998,"15min":0.872},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.03,"5min":1.216,"15min":1.216},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.90200000000000002,"15min":0.92900000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0429999999999999,"5min":1.329,"15min":1.329},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.877,"5min":0.88500000000000001,"15min":0.88700000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Nov 16 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.86299999999999999}]},{"osd":7,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.97099999999999997,"15min":0.98299999999999998},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.02,"15min":1.038},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":0.83499999999999996}]},{"osd":8,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.006,"15min":1.0349999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.1080000000000001,"5min":1.357,"15min":1.357},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.98799999999999999,"15min":0.98999999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.6429999999999998,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.874,"15min":0.878},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0649999999999999,"5min":1.141,"15min":1.141},"last":0.86899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103206,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":978944,"data_stored":471592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.58099999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.748,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89600000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.63400000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.46899999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68799999999999994,"15min":0.67000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":4.5149999999999997,"5min":4.5149999999999997,"15min":4.5149999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84599999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.77700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.91700000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.43099999999999999}]},{"osd":6,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78500000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76500000000000001,"15min":0.75},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68700000000000006,"15min":0.69799999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.74199999999999999}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77700000000000002,"15min":0.76900000000000002},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":0.85299999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7040,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738112,"statfs":{"total":95995035648,"available":95987826688,"internally_reserved":0,"allocated":1564672,"data_stored":1061916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.53600000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81599999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.66400000000000003}]},{"osd":3,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57299999999999995,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77800000000000002,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.94599999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.501,"15min":0.501},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.499},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.503,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.44400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.45900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.83799999999999997}]},{"osd":6,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":1.069,"15min":1.069},"last":0.64300000000000002}]},{"osd":8,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.626,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.51200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.77800000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.55500000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.87},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.93700000000000006}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6396,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738756,"statfs":{"total":95995035648,"available":95988486144,"internally_reserved":0,"allocated":970752,"data_stored":471056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.35399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86299999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79000000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69399999999999995,"15min":0.68100000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91000000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80200000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85599999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.626}]},{"osd":4,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.75800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.111,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73099999999999998,"15min":0.73499999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.85899999999999999,"5min":1.044,"15min":1.044},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.78100000000000003,"15min":0.78900000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.95099999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.754,"15min":0.755},"min":{"1min":0.52300000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94999999999999996,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:16 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.81599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78800000000000003,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.66000000000000003}]},{"osd":8,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85899999999999999,"15min":0.86599999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.032,"5min":1.175,"15min":1.175},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.84399999999999997,"15min":0.81200000000000006},"min":{"1min":0.59399999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.5839999999999996,"5min":5.5839999999999996,"15min":5.5839999999999996},"last":0.76400000000000001}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.91500000000000004,"15min":0.92500000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.91100000000000003}]},{"osd":10,"last update":"Thu Nov 16 23:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.87},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.70599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7384,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737768,"statfs":{"total":95995035648,"available":95987474432,"internally_reserved":0,"allocated":999424,"data_stored":492141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.50700000000000001},"min":{"1min":0.48899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89200000000000002,"5min":0.998,"15min":0.998},"last":0.63},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80700000000000005,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Nov 16 23:59:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.56399999999999995,"15min":0.54400000000000004},"min":{"1min":0.42099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.45200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.41399999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78200000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.58899999999999997,"15min":0.55100000000000005},"min":{"1min":0.45900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86799999999999999,"5min":1.028,"15min":1.028},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.496},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.432,"15min":0.40100000000000002},"min":{"1min":0.34300000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.443}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.55600000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.877,"5min":1.103,"15min":1.103},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.38,"15min":0.38},"max":{"1min":0.83099999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.745}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57999999999999996,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.81799999999999995}]},{"osd":8,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.59799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66000000000000003,"15min":0.64000000000000001},"min":{"1min":0.59299999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.71599999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:02 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73899999999999999,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.96599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79400000000000004,"15min":0.78900000000000003},"min":{"1min":0.625,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.90300000000000002}]},{"osd":10,"last update":"Thu Nov 16 23:58:36 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.875},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.77300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7364,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737788,"statfs":{"total":95995035648,"available":95987494912,"internally_reserved":0,"allocated":978944,"data_stored":471660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 23:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57999999999999996,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85999999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59599999999999997,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82599999999999996,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.55900000000000005},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.64100000000000001,"15min":0.60799999999999998},"min":{"1min":0.46400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0169999999999999,"5min":1.071,"15min":1.071},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.51900000000000002,"15min":0.503},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.499},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.35699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:59:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.68100000000000005,"15min":0.625},"min":{"1min":0.47699999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.754},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Nov 16 23:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.61199999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67200000000000004,"15min":0.65500000000000003},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:59:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67900000000000005,"15min":0.67700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.96299999999999997,"5min":1.254,"15min":1.254},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.73599999999999999,"15min":0.70899999999999996},"min":{"1min":0.45900000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.876}]},{"osd":7,"last update":"Thu Nov 16 23:58:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.093,"5min":1.135,"15min":1.135},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.85399999999999998,"15min":0.86499999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.80900000000000005,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1279999999999999,"5min":1.276,"15min":1.276},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":0.88},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.93000000000000005,"15min":0.93200000000000005},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.98099999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.042}]},{"osd":11,"last update":"Thu Nov 16 23:59:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.018}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7368,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737784,"statfs":{"total":95995035648,"available":95987490816,"internally_reserved":0,"allocated":983040,"data_stored":474794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.72399999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.50600000000000001},"min":{"1min":0.38200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:58:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.42499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Nov 16 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.435,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.752,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55000000000000004,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.72099999999999997}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Nov 16 23:58:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Nov 16 23:58:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.63},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73299999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.78200000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.749}]},{"osd":8,"last update":"Thu Nov 16 23:58:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64600000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72499999999999998,"15min":0.71299999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.67500000000000004}]},{"osd":10,"last update":"Thu Nov 16 23:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":470919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.40899999999999997,"15min":0.38400000000000001},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.42599999999999999,"15min":0.40000000000000002},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57599999999999996,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.625,"15min":0.58899999999999997},"min":{"1min":0.55800000000000005,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:58:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.69499999999999995,"15min":0.61499999999999999},"min":{"1min":0.499,"5min":0.37,"15min":0.37},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Thu Nov 16 23:59:12 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.68000000000000005,"15min":0.622},"min":{"1min":0.50900000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.042,"5min":1.405,"15min":1.405},"last":0.87},{"interface":"front","average":{"1min":0.747,"5min":0.65800000000000003,"15min":0.61899999999999999},"min":{"1min":0.54500000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Nov 16 23:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69599999999999995,"15min":0.63900000000000001},"min":{"1min":0.51400000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.1220000000000001,"5min":1.492,"15min":1.492},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68999999999999995,"15min":0.64600000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.212,"15min":1.212},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Nov 16 23:58:55 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58999999999999997,"15min":0.56999999999999995},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.72499999999999998,"15min":0.68899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.98599999999999999}]},{"osd":7,"last update":"Thu Nov 16 23:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.79800000000000004},"min":{"1min":0.66300000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1739999999999999,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77600000000000002,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1419999999999999,"5min":1.256,"15min":1.256},"last":0.92100000000000004}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.90600000000000003,"15min":0.86799999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":7.173,"5min":7.173,"15min":7.173},"last":0.84499999999999997}]},{"osd":9,"last update":"Thu Nov 16 23:59:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81599999999999995,"15min":0.82899999999999996},"min":{"1min":0.55900000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2030000000000001,"5min":1.23,"15min":1.23},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.998,"15min":1.014},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.1399999999999999,"5min":1.617,"15min":1.617},"last":0.95899999999999996}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":7.1600000000000001,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":1.036},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.97399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6532,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738620,"statfs":{"total":95995035648,"available":95988346880,"internally_reserved":0,"allocated":978944,"data_stored":471562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.27700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65200000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38600000000000001,"15min":0.36699999999999999},"min":{"1min":0.29199999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56299999999999994,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.379,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.38,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.753,"5min":0.872,"15min":0.872},"last":0.62}]},{"osd":3,"last update":"Thu Nov 16 23:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.58999999999999997},"min":{"1min":0.53900000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85299999999999998,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64800000000000002,"15min":0.62},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":1.105,"15min":1.105},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Nov 16 23:59:31 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Nov 16 23:58:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.62,"15min":0.59999999999999998},"min":{"1min":0.46200000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80800000000000005,"5min":0.997,"15min":0.997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.45600000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.875,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.71899999999999997}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84399999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.755},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82699999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81899999999999995}]},{"osd":9,"last update":"Thu Nov 16 23:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72299999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.91000000000000003,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.56799999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.90400000000000003,"5min":1.05,"15min":1.05},"last":0.79100000000000004}]},{"osd":10,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.872},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.78000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7188,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737964,"statfs":{"total":95995035648,"available":95987675136,"internally_reserved":0,"allocated":1585152,"data_stored":1062431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52000000000000002},"min":{"1min":0.42199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.753},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.753,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40400000000000003,"15min":0.38800000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55700000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.38700000000000001},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.442}]},{"osd":3,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.503},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.47099999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.55300000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81299999999999994,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.44,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.623}]},{"osd":6,"last update":"Thu Nov 16 23:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63100000000000001,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55700000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63700000000000001,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.746},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.44500000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60199999999999998}]},{"osd":8,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82499999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.58399999999999996}]},{"osd":9,"last update":"Thu Nov 16 23:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.55500000000000005,"5min":0.497,"15min":0.497},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.92900000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68500000000000005}]},{"osd":10,"last update":"Thu Nov 16 23:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T23:59:38.738 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-16T23:59:38.741 INFO:tasks.cephadm:Setting up client nodes... 2023-11-16T23:59:38.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-16T23:59:39.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:38 smithi023 ceph-mon[97824]: mgrmap e17: x(active, since 8m), standbys: z, y 2023-11-16T23:59:39.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:38 smithi023 ceph-mon[97824]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:39.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:38 smithi173 ceph-mon[131534]: mgrmap e17: x(active, since 8m), standbys: z, y 2023-11-16T23:59:39.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:38 smithi173 ceph-mon[131534]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:39.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:38 smithi156 ceph-mon[131105]: mgrmap e17: x(active, since 8m), standbys: z, y 2023-11-16T23:59:39.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:38 smithi156 ceph-mon[131105]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:39.604 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:40.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:39 smithi023 ceph-mon[97824]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:40.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:39 smithi023 ceph-mon[97824]: pgmap v322: 129 pgs: 129 active+clean; 584 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2023-11-16T23:59:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:39 smithi156 ceph-mon[131105]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:39 smithi156 ceph-mon[131105]: pgmap v322: 129 pgs: 129 active+clean; 584 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2023-11-16T23:59:40.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:39 smithi173 ceph-mon[131534]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T23:59:40.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:39 smithi173 ceph-mon[131534]: pgmap v322: 129 pgs: 129 active+clean; 584 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.244+0000 7f5c60999700 1 -- 172.21.15.23:0/349555456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1014f0 msgr2=0x7f5c5c101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.244+0000 7f5c60999700 1 --2- 172.21.15.23:0/349555456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c101940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009990 tx=0x7f5c50009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 -- 172.21.15.23:0/349555456 shutdown_connections 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 --2- 172.21.15.23:0/349555456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 0x7f5c5c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 --2- 172.21.15.23:0/349555456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 --2- 172.21.15.23:0/349555456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c101940 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 -- 172.21.15.23:0/349555456 >> 172.21.15.23:0/349555456 conn(0x7f5c5c0fb900 msgr2=0x7f5c5c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 -- 172.21.15.23:0/349555456 shutdown_connections 2023-11-16T23:59:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.245+0000 7f5c60999700 1 -- 172.21.15.23:0/349555456 wait complete. 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 Processor -- start 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 -- start start 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 0x7f5c5c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19dc00 con 0x7f5c5c1002f0 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19dd40 con 0x7f5c5c1014f0 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c60999700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19ded0 con 0x7f5c5c101e80 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.246+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48826/0 (socket says 172.21.15.23:48826) 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 -- 172.21.15.23:0/1335373898 learned_addr learned my addr 172.21.15.23:0/1335373898 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c59d9b700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5ad9d700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 0x7f5c5c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 -- 172.21.15.23:0/1335373898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 msgr2=0x7f5c5c19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 0x7f5c5c19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 -- 172.21.15.23:0/1335373898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1014f0 msgr2=0x7f5c5c1965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c1965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 -- 172.21.15.23:0/1335373898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500095f0 con 0x7f5c5c1002f0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c5a59c700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00e9a0 tx=0x7f5c4c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.247+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c009790 con 0x7f5c5c1002f0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.248+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c4c00c810 con 0x7f5c5c1002f0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.248+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c10bcd0 con 0x7f5c5c1002f0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.248+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c003c70 con 0x7f5c5c1002f0 2023-11-16T23:59:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.248+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c10bb40 con 0x7f5c5c1002f0 2023-11-16T23:59:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.249+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c066ea0 con 0x7f5c5c1002f0 2023-11-16T23:59:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.251+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5c4c00cc80 con 0x7f5c5c1002f0 2023-11-16T23:59:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.252+0000 7f5c4b7fe700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 0x7f5c440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.252+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5c4c014070 con 0x7f5c5c1002f0 2023-11-16T23:59:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.252+0000 7f5c59d9b700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 0x7f5c440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.253+0000 7f5c59d9b700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 0x7f5c440a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000b2e0 tx=0x7f5c50005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.256+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5c4c096210 con 0x7f5c5c1002f0 2023-11-16T23:59:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.543+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f5c5c107240 con 0x7f5c5c1002f0 2023-11-16T23:59:41.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.549+0000 7f5c4b7fe700 1 -- 172.21.15.23:0/1335373898 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f5c4c003dd0 con 0x7f5c5c1002f0 2023-11-16T23:59:41.550 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2023-11-16T23:59:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDtrFZl1tx0IBAAe8EBdf59X9Jb5dejg+vnpg== 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 msgr2=0x7f5c440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 0x7f5c440a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000b2e0 tx=0x7f5c50005fb0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 msgr2=0x7f5c5c196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00e9a0 tx=0x7f5c4c00ecb0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 shutdown_connections 2023-11-16T23:59:41.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5c440a11e0 0x7f5c440a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c5c101e80 0x7f5c5c19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c5c1002f0 0x7f5c5c196070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 --2- 172.21.15.23:0/1335373898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c1014f0 0x7f5c5c1965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.551+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 >> 172.21.15.23:0/1335373898 conn(0x7f5c5c0fb900 msgr2=0x7f5c5c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.552+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 shutdown_connections 2023-11-16T23:59:41.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:41.552+0000 7f5c60999700 1 -- 172.21.15.23:0/1335373898 wait complete. 2023-11-16T23:59:42.113 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-16T23:59:42.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-16T23:59:42.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-16T23:59:42.217 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-16T23:59:42.217 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-16T23:59:42.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-16T23:59:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:42 smithi023 ceph-mon[97824]: pgmap v323: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-11-16T23:59:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:42 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T23:59:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:42 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:42 smithi156 ceph-mon[131105]: pgmap v323: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:42 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:42 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:42 smithi173 ceph-mon[131534]: pgmap v323: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:42 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T23:59:42.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:42 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1335373898' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T23:59:43.117 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:44.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:44 smithi023 ceph-mon[97824]: pgmap v324: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T23:59:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:44 smithi156 ceph-mon[131105]: pgmap v324: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T23:59:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:44 smithi173 ceph-mon[131534]: pgmap v324: 129 pgs: 129 active+clean; 618 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-11-16T23:59:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.632+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/2035901399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100ee0 msgr2=0x7fe7dc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.632+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/2035901399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc104eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0099c0 tx=0x7fe7cc009cd0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/2035901399 shutdown_connections 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/2035901399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/2035901399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100540 0x7fe7dc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/2035901399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc104eb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/2035901399 >> 172.21.15.23:0/2035901399 conn(0x7fe7dc0fbdc0 msgr2=0x7fe7dc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/2035901399 shutdown_connections 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.633+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/2035901399 wait complete. 2023-11-16T23:59:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 Processor -- start 2023-11-16T23:59:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 -- start start 2023-11-16T23:59:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1a4840 con 0x7fe7dc100540 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1a49b0 con 0x7fe7dc100ee0 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.634+0000 7fe7e3f4b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1a4b70 con 0x7fe7dc1055e0 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e1ce7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:44790/0 (socket says 172.21.15.23:44790) 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 -- 172.21.15.23:0/4289151689 learned_addr learned my addr 172.21.15.23:0/4289151689 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e1ce7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48846/0 (socket says 172.21.15.23:48846) 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e14e6700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 -- 172.21.15.23:0/4289151689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100ee0 msgr2=0x7fe7dc073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 -- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 msgr2=0x7fe7dc072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 -- 172.21.15.23:0/4289151689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc0095f0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e1ce7700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.635+0000 7fe7e24e8700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400ef30 tx=0x7fe7d400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.636+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4018070 con 0x7fe7dc1055e0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.636+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7dc110360 con 0x7fe7dc1055e0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.636+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7d400f040 con 0x7fe7dc1055e0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.636+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc1101d0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.636+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400ccb0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.637+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c0005320 con 0x7fe7dc1055e0 2023-11-16T23:59:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.639+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe7d4004ad0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.640+0000 7fe7d2ffd700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 0x7fe7c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.641+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7d40cc5b0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.641+0000 7fe7e1ce7700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 0x7fe7c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.641+0000 7fe7e1ce7700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 0x7fe7c80a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8005950 tx=0x7fe7d800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.645+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe7d4099470 con 0x7fe7dc1055e0 2023-11-16T23:59:44.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.905+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe7c0005190 con 0x7fe7dc1055e0 2023-11-16T23:59:44.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.910+0000 7fe7d2ffd700 1 -- 172.21.15.23:0/4289151689 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260121 (secure 0 0 0) 0x7fe7d40092e0 con 0x7fe7dc1055e0 2023-11-16T23:59:44.916 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:44.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 msgr2=0x7fe7c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 0x7fe7c80a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8005950 tx=0x7fe7d800b410 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 msgr2=0x7fe7dc0738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400ef30 tx=0x7fe7d400c3a0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 shutdown_connections 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a1260 0x7fe7c80a3710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7dc1055e0 0x7fe7dc0738b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7dc100540 0x7fe7dc072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 --2- 172.21.15.23:0/4289151689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7dc100ee0 0x7fe7dc073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.914+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 >> 172.21.15.23:0/4289151689 conn(0x7fe7dc0fbdc0 msgr2=0x7fe7dc0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.915+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 shutdown_connections 2023-11-16T23:59:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:44.915+0000 7fe7e3f4b700 1 -- 172.21.15.23:0/4289151689 wait complete. 2023-11-16T23:59:45.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:45 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4289151689' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T23:59:45.414 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":499313593},{"type":"v1","addr":"172.21.15.23:6801","nonce":499313593}]},"active_addr":"172.21.15.23:6801/499313593","active_change":"2023-11-16T23:51:30.957922+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":4223790019}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1775702876}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1032636545}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":19269155}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2153551661}]}]} 2023-11-16T23:59:45.433 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-16T23:59:45.433 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-16T23:59:45.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T23:59:45.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:45 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4289151689' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T23:59:45.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:45 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4289151689' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T23:59:46.290 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:46 smithi023 ceph-mon[97824]: pgmap v325: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.7 KiB/s wr, 15 op/s 2023-11-16T23:59:46.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:46 smithi173 ceph-mon[131534]: pgmap v325: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.7 KiB/s wr, 15 op/s 2023-11-16T23:59:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:46 smithi156 ceph-mon[131105]: pgmap v325: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.7 KiB/s wr, 15 op/s 2023-11-16T23:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.667+0000 7f3cfb119700 1 -- 172.21.15.23:0/4065048531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf41055e0 msgr2=0x7f3cf410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.667+0000 7f3cfb119700 1 --2- 172.21.15.23:0/4065048531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf41055e0 0x7f3cf410d160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00b0d0 tx=0x7f3cec00b3e0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 -- 172.21.15.23:0/4065048531 shutdown_connections 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 --2- 172.21.15.23:0/4065048531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf41055e0 0x7f3cf410d160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 --2- 172.21.15.23:0/4065048531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100540 0x7f3cf4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 --2- 172.21.15.23:0/4065048531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf4100ee0 0x7f3cf4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 -- 172.21.15.23:0/4065048531 >> 172.21.15.23:0/4065048531 conn(0x7f3cf40fbdc0 msgr2=0x7f3cf40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 -- 172.21.15.23:0/4065048531 shutdown_connections 2023-11-16T23:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.668+0000 7f3cfb119700 1 -- 172.21.15.23:0/4065048531 wait complete. 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 Processor -- start 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 -- start start 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100ee0 0x7f3cf419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf41055e0 0x7f3cf4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4198b60 con 0x7f3cf4100540 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4198ca0 con 0x7f3cf4100ee0 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.669+0000 7f3cfb119700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4198e60 con 0x7f3cf41055e0 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48870/0 (socket says 172.21.15.23:48870) 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 -- 172.21.15.23:0/2061281388 learned_addr learned my addr 172.21.15.23:0/2061281388 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf96b6700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf41055e0 0x7f3cf4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf3fff700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100ee0 0x7f3cf419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 -- 172.21.15.23:0/2061281388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf41055e0 msgr2=0x7f3cf4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf41055e0 0x7f3cf4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 -- 172.21.15.23:0/2061281388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100ee0 msgr2=0x7f3cf419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100ee0 0x7f3cf419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 -- 172.21.15.23:0/2061281388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8009520 con 0x7f3cf4100540 2023-11-16T23:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf8eb5700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800eb30 tx=0x7f3ce800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.670+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8009880 con 0x7f3cf4100540 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.671+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ce80044e0 con 0x7f3cf4100540 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.671+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec009d00 con 0x7f3cf4100540 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.671+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8003e90 con 0x7f3cf4100540 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.671+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4110360 con 0x7f3cf4100540 2023-11-16T23:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.672+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf41101d0 con 0x7f3cf4100540 2023-11-16T23:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.674+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3ce8004000 con 0x7f3cf4100540 2023-11-16T23:59:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.676+0000 7f3cf1ffb700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 0x7f3ce40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.676+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3ce8014070 con 0x7f3cf4100540 2023-11-16T23:59:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.676+0000 7f3cf3fff700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 0x7f3ce40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.677+0000 7f3cf3fff700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 0x7f3ce40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4199cf0 tx=0x7f3ce0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:47.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.680+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3ce80964e0 con 0x7f3cf4100540 2023-11-16T23:59:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.888+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cf41104f0 con 0x7f3cf4100540 2023-11-16T23:59:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.889+0000 7f3cf1ffb700 1 -- 172.21.15.23:0/2061281388 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f3ce8099b00 con 0x7f3cf4100540 2023-11-16T23:59:47.890 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:47.890 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 msgr2=0x7f3ce40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 0x7f3ce40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4199cf0 tx=0x7f3ce0009260 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 msgr2=0x7f3cf419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800eb30 tx=0x7f3ce800ee40 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 shutdown_connections 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3ce40a11e0 0x7f3ce40a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3cf41055e0 0x7f3cf4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf4100540 0x7f3cf419e420 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 --2- 172.21.15.23:0/2061281388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf4100ee0 0x7f3cf419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 >> 172.21.15.23:0/2061281388 conn(0x7f3cf40fbdc0 msgr2=0x7f3cf410b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 shutdown_connections 2023-11-16T23:59:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:47.892+0000 7f3cfb119700 1 -- 172.21.15.23:0/2061281388 wait complete. 2023-11-16T23:59:48.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:48 smithi023 ceph-mon[97824]: pgmap v326: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2023-11-16T23:59:48.276 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:48 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2061281388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:48.290 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-16T23:59:48.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T23:59:48.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:48 smithi173 ceph-mon[131534]: pgmap v326: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2023-11-16T23:59:48.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:48 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2061281388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:48 smithi156 ceph-mon[131105]: pgmap v326: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.4 KiB/s wr, 11 op/s 2023-11-16T23:59:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:48 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2061281388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:49.151 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:49 smithi023 ceph-mon[97824]: pgmap v327: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2023-11-16T23:59:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:49 smithi173 ceph-mon[131534]: pgmap v327: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2023-11-16T23:59:50.111 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:49 smithi156 ceph-mon[131105]: pgmap v327: 129 pgs: 129 active+clean; 627 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 -- 172.21.15.23:0/1969275686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 msgr2=0x7f15cc100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 --2- 172.21.15.23:0/1969275686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc100910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f15c80099c0 tx=0x7f15c8009cd0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 -- 172.21.15.23:0/1969275686 shutdown_connections 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 --2- 172.21.15.23:0/1969275686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 0x7f15cc10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 --2- 172.21.15.23:0/1969275686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc100910 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 --2- 172.21.15.23:0/1969275686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 -- 172.21.15.23:0/1969275686 >> 172.21.15.23:0/1969275686 conn(0x7f15cc0fbdc0 msgr2=0x7f15cc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 -- 172.21.15.23:0/1969275686 shutdown_connections 2023-11-16T23:59:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.699+0000 7f15d40dd700 1 -- 172.21.15.23:0/1969275686 wait complete. 2023-11-16T23:59:50.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.700+0000 7f15d40dd700 1 Processor -- start 2023-11-16T23:59:50.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 -- start start 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 0x7f15cc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc19b3b0 con 0x7f15cc100540 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc19b520 con 0x7f15cc1055e0 2023-11-16T23:59:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d40dd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc19b6e0 con 0x7f15cc100ee0 2023-11-16T23:59:50.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.701+0000 7f15d1e79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:50.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48898/0 (socket says 172.21.15.23:48898) 2023-11-16T23:59:50.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 -- 172.21.15.23:0/2027351532 learned_addr learned my addr 172.21.15.23:0/2027351532 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:50.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1678700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d267a700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 0x7f15cc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 -- 172.21.15.23:0/2027351532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 msgr2=0x7f15cc196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 -- 172.21.15.23:0/2027351532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 msgr2=0x7f15cc19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 0x7f15cc19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 -- 172.21.15.23:0/2027351532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c80095f0 con 0x7f15cc100540 2023-11-16T23:59:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1678700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15d1e79700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009990 tx=0x7f15c800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.702+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c801d070 con 0x7f15cc100540 2023-11-16T23:59:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.703+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15c8005030 con 0x7f15cc100540 2023-11-16T23:59:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.703+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c8004a50 con 0x7f15cc100540 2023-11-16T23:59:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.703+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc110370 con 0x7f15cc100540 2023-11-16T23:59:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.703+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15cc1101e0 con 0x7f15cc100540 2023-11-16T23:59:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.704+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc110500 con 0x7f15cc100540 2023-11-16T23:59:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.706+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f15c8005430 con 0x7f15cc100540 2023-11-16T23:59:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.707+0000 7f15c2ffd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 0x7f15b80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.707+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f15c80cd990 con 0x7f15cc100540 2023-11-16T23:59:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.707+0000 7f15d1678700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 0x7f15b80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.708+0000 7f15d1678700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 0x7f15b80a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f15cc197a10 tx=0x7f15bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.711+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f15c8096dc0 con 0x7f15cc100540 2023-11-16T23:59:50.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.920+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15cc1a3470 con 0x7f15cc100540 2023-11-16T23:59:50.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.922+0000 7f15c2ffd700 1 -- 172.21.15.23:0/2027351532 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f15c809a3e0 con 0x7f15cc100540 2023-11-16T23:59:50.923 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-16T23:59:50.923 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-16T23:58:51.233842+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T23:59:50.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.924+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 msgr2=0x7f15b80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:50.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.924+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 0x7f15b80a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f15cc197a10 tx=0x7f15bc009260 comp rx=0 tx=0).stop 2023-11-16T23:59:50.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.924+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 msgr2=0x7f15cc1963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:50.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.924+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f15c8009990 tx=0x7f15c800f690 comp rx=0 tx=0).stop 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 shutdown_connections 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc1055e0 0x7f15cc19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15cc100540 0x7f15cc1963f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f15b80a1260 0x7f15b80a3710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 --2- 172.21.15.23:0/2027351532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15cc100ee0 0x7f15cc196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 >> 172.21.15.23:0/2027351532 conn(0x7f15cc0fbdc0 msgr2=0x7f15cc0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 shutdown_connections 2023-11-16T23:59:50.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:50.925+0000 7f15d40dd700 1 -- 172.21.15.23:0/2027351532 wait complete. 2023-11-16T23:59:51.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:50 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2027351532' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:50 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-16T23:59:51.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-16T23:59:51.358 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-16T23:59:51.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-16T23:59:51.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-16T23:59:51.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-16T23:59:51.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-16T23:59:51.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-16T23:59:51.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-11-16T23:59:51.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-11-16T23:59:51.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-11-16T23:59:51.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-11-16T23:59:51.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:50 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2027351532' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:50 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:50 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2027351532' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:50 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T23:59:51.996 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:51 smithi023 ceph-mon[97824]: pgmap v328: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-11-16T23:59:52.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:51 smithi156 ceph-mon[131105]: pgmap v328: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-11-16T23:59:52.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:51 smithi173 ceph-mon[131534]: pgmap v328: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-11-16T23:59:53.532 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:53.668 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:53.684 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:53.966 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:53.968 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:54.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:54 smithi023 ceph-mon[97824]: pgmap v329: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T23:59:54.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.349 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:54 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:54 smithi173 ceph-mon[131534]: pgmap v329: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:54 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:54 smithi156 ceph-mon[131105]: pgmap v329: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-11-16T23:59:54.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:54.612 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:54 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-16T23:59:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:55.258 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:55.283 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:55.521 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:56 smithi023 ceph-mon[97824]: pgmap v330: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T23:59:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:56 smithi156 ceph-mon[131105]: pgmap v330: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T23:59:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:56 smithi173 ceph-mon[131534]: pgmap v330: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T23:59:57.388 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T23:59:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.070+0000 7f0ec2713700 1 -- 172.21.15.23:0/559074645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc10a890 msgr2=0x7f0ebc10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.070+0000 7f0ec2713700 1 --2- 172.21.15.23:0/559074645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc10a890 0x7f0ebc10ccd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400b0a0 tx=0x7f0eb400b3b0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 -- 172.21.15.23:0/559074645 shutdown_connections 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 --2- 172.21.15.23:0/559074645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc10a890 0x7f0ebc10ccd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 --2- 172.21.15.23:0/559074645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc101540 0x7f0ebc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 --2- 172.21.15.23:0/559074645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ebc101e50 0x7f0ebc10a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 -- 172.21.15.23:0/559074645 >> 172.21.15.23:0/559074645 conn(0x7f0ebc0770a0 msgr2=0x7f0ebc0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 -- 172.21.15.23:0/559074645 shutdown_connections 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.071+0000 7f0ec2713700 1 -- 172.21.15.23:0/559074645 wait complete. 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.072+0000 7f0ec2713700 1 Processor -- start 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 -- start start 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc101540 0x7f0ebc198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ebc101e50 0x7f0ebc198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 0x7f0ebc19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc19d540 con 0x7f0ebc101e50 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc19d6b0 con 0x7f0ebc10a890 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.073+0000 7f0ec2713700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc19d870 con 0x7f0ebc101540 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 0x7f0ebc19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 0x7f0ebc19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:33192/0 (socket says 172.21.15.23:33192) 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 -- 172.21.15.23:0/1477091475 learned_addr learned my addr 172.21.15.23:0/1477091475 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ebb7fe700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ebc101e50 0x7f0ebc198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 -- 172.21.15.23:0/1477091475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc101540 msgr2=0x7f0ebc198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ebc101540 0x7f0ebc198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 -- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ebc101e50 msgr2=0x7f0ebc198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ebc101e50 0x7f0ebc198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.074+0000 7f0ec0cb0700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4009d00 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.075+0000 7f0ec0cb0700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 0x7f0ebc19cf30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40062a0 tx=0x7f0eb4004d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.075+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb400e040 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.075+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc1007a0 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.076+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0eb40094d0 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.076+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb40045b0 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.076+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc100610 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.077+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ebc061270 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.080+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0eb4004710 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.082+0000 7f0eb97fa700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0ea40a1260 0x7f0ea40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0eb40ce7d0 con 0x7f0ebc10a890 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0ebbfff700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0ea40a1260 0x7f0ea40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0eb97fa700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] conn(0x7f0ea40abdd0 0x7f0ea40ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ea40ae8d0 con 0x7f0ea40abdd0 2023-11-16T23:59:58.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0eb40ceb40 con 0x7f0ebc10a890 2023-11-16T23:59:58.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0ebb7fe700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] conn(0x7f0ea40abdd0 0x7f0ea40ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:58.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.083+0000 7f0ebbfff700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0ea40a1260 0x7f0ea40a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0eac005950 tx=0x7f0eac0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:58.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.084+0000 7f0ebb7fe700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] conn(0x7f0ea40abdd0 0x7f0ea40ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:58.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.085+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== osd.0 v2:172.21.15.23:6802/2230726464 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0ea40ae8d0 con 0x7f0ea40abdd0 2023-11-16T23:59:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.113+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 --> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ebc100930 con 0x7f0ea40abdd0 2023-11-16T23:59:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.113+0000 7f0eb97fa700 1 -- 172.21.15.23:0/1477091475 <== osd.0 v2:172.21.15.23:6802/2230726464 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0ebc100930 con 0x7f0ea40abdd0 2023-11-16T23:59:58.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] conn(0x7f0ea40abdd0 msgr2=0x7f0ea40ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] conn(0x7f0ea40abdd0 0x7f0ea40ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0ea40a1260 msgr2=0x7f0ea40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0ea40a1260 0x7f0ea40a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0eac005950 tx=0x7f0eac0058e0 comp rx=0 tx=0).stop 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 msgr2=0x7f0ebc19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 --2- 172.21.15.23:0/1477091475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc10a890 0x7f0ebc19cf30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40062a0 tx=0x7f0eb4004d30 comp rx=0 tx=0).stop 2023-11-16T23:59:58.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec0cb0700 1 -- 172.21.15.23:0/1477091475 reap_dead start 2023-11-16T23:59:58.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 shutdown_connections 2023-11-16T23:59:58.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 >> 172.21.15.23:0/1477091475 conn(0x7f0ebc0770a0 msgr2=0x7f0ebc1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:58.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 shutdown_connections 2023-11-16T23:59:58.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:58.114+0000 7f0ec2713700 1 -- 172.21.15.23:0/1477091475 wait complete. 2023-11-16T23:59:58.213 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:58 smithi023 ceph-mon[97824]: pgmap v331: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T23:59:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:58 smithi173 ceph-mon[131534]: pgmap v331: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T23:59:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:58 smithi156 ceph-mon[131105]: pgmap v331: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T23:59:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 -- 172.21.15.23:0/934911582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941086b0 msgr2=0x7f9f94108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/934911582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941086b0 0x7f9f94108a80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009990 tx=0x7f9f90009ca0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 -- 172.21.15.23:0/934911582 shutdown_connections 2023-11-16T23:59:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/934911582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 0x7f9f9410f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/934911582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941086b0 0x7f9f94108a80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/934911582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941025c0 0x7f9f94102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.180+0000 7f9f9b40b700 1 -- 172.21.15.23:0/934911582 >> 172.21.15.23:0/934911582 conn(0x7f9f940fe100 msgr2=0x7f9f94100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.181+0000 7f9f9b40b700 1 -- 172.21.15.23:0/934911582 shutdown_connections 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.181+0000 7f9f9b40b700 1 -- 172.21.15.23:0/934911582 wait complete. 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.182+0000 7f9f9b40b700 1 Processor -- start 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.182+0000 7f9f9b40b700 1 -- start start 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941025c0 0x7f9f94198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 0x7f9f94198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941086b0 0x7f9f9419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9419d510 con 0x7f9f941025c0 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9419d680 con 0x7f9f94102f50 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f9b40b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9419d840 con 0x7f9f941086b0 2023-11-16T23:59:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f989a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 0x7f9f94198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f989a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 0x7f9f94198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:33200/0 (socket says 172.21.15.23:33200) 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f989a6700 1 -- 172.21.15.23:0/4024732814 learned_addr learned my addr 172.21.15.23:0/4024732814 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.183+0000 7f9f991a7700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941025c0 0x7f9f94198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f999a8700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941086b0 0x7f9f9419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f991a7700 1 -- 172.21.15.23:0/4024732814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941086b0 msgr2=0x7f9f9419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f991a7700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941086b0 0x7f9f9419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f991a7700 1 -- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 msgr2=0x7f9f94198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f991a7700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f94102f50 0x7f9f94198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.184+0000 7f9f991a7700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f900095f0 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.185+0000 7f9f999a8700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f941086b0 0x7f9f9419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.185+0000 7f9f991a7700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941025c0 0x7f9f94198550 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f90005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.186+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9001d070 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.186+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f94112600 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.186+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f90021410 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.186+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90017650 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.186+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f94112470 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.188+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f78000ff0 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.192+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9f9000fb10 con 0x7f9f941025c0 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.193+0000 7f9f8a7fc700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f800a1260 0x7f9f800a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.193+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9f900ce7b0 con 0x7f9f941025c0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.194+0000 7f9f8a7fc700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] conn(0x7f9f800abdd0 0x7f9f800ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.194+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f800ae8d0 con 0x7f9f800abdd0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.194+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f900cebc0 con 0x7f9f941025c0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.194+0000 7f9f989a6700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f800a1260 0x7f9f800a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.194+0000 7f9f999a8700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] conn(0x7f9f800abdd0 0x7f9f800ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.195+0000 7f9f989a6700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f800a1260 0x7f9f800a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9f94199b70 tx=0x7f9f84009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.195+0000 7f9f999a8700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] conn(0x7f9f800abdd0 0x7f9f800ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.196+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== osd.4 v2:172.21.15.156:6808/1251331941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9f800ae8d0 con 0x7f9f800abdd0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.215+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 --> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f78002d70 con 0x7f9f800abdd0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f8a7fc700 1 -- 172.21.15.23:0/4024732814 <== osd.4 v2:172.21.15.156:6808/1251331941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9f78002d70 con 0x7f9f800abdd0 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] conn(0x7f9f800abdd0 msgr2=0x7f9f800ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] conn(0x7f9f800abdd0 0x7f9f800ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T23:59:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f800a1260 msgr2=0x7f9f800a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9f800a1260 0x7f9f800a3710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9f94199b70 tx=0x7f9f84009260 comp rx=0 tx=0).stop 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941025c0 msgr2=0x7f9f94198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 --2- 172.21.15.23:0/4024732814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f941025c0 0x7f9f94198550 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f90005740 comp rx=0 tx=0).stop 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f999a8700 1 -- 172.21.15.23:0/4024732814 reap_dead start 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 shutdown_connections 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.216+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 >> 172.21.15.23:0/4024732814 conn(0x7f9f940fe100 msgr2=0x7f9f940ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.217+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 shutdown_connections 2023-11-16T23:59:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-16T23:59:59.217+0000 7f9f9b40b700 1 -- 172.21.15.23:0/4024732814 wait complete. 2023-11-16T23:59:59.852 INFO:teuthology.orchestra.run.smithi023.stdout:34359738451 2023-11-16T23:59:59.853 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-16T23:59:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 16 23:59:59 smithi023 ceph-mon[97824]: pgmap v332: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-17T00:00:00.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 16 23:59:59 smithi156 ceph-mon[131105]: pgmap v332: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-17T00:00:00.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 16 23:59:59 smithi173 ceph-mon[131534]: pgmap v332: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-17T00:00:00.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:00 smithi023 ceph-mon[97824]: overall HEALTH_OK 2023-11-17T00:00:01.001 INFO:teuthology.orchestra.run.smithi023.stdout:141733920830 2023-11-17T00:00:01.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-17T00:00:01.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:00 smithi173 ceph-mon[131534]: overall HEALTH_OK 2023-11-17T00:00:01.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:00 smithi156 ceph-mon[131105]: overall HEALTH_OK 2023-11-17T00:00:01.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 -- 172.21.15.23:0/3033509846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 msgr2=0x7f71ac100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 --2- 172.21.15.23:0/3033509846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac100740 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f71a80099c0 tx=0x7f71a8009cd0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 -- 172.21.15.23:0/3033509846 shutdown_connections 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 --2- 172.21.15.23:0/3033509846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71ac105410 0x7f71ac10cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 --2- 172.21.15.23:0/3033509846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac100740 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 --2- 172.21.15.23:0/3033509846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac100d10 0x7f71ac104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 -- 172.21.15.23:0/3033509846 >> 172.21.15.23:0/3033509846 conn(0x7f71ac0fbbd0 msgr2=0x7f71ac0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:01.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 -- 172.21.15.23:0/3033509846 shutdown_connections 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.899+0000 7f71b2355700 1 -- 172.21.15.23:0/3033509846 wait complete. 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.900+0000 7f71b2355700 1 Processor -- start 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.900+0000 7f71b2355700 1 -- start start 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac100d10 0x7f71ac198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71ac105410 0x7f71ac19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19d410 con 0x7f71ac100370 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19d580 con 0x7f71ac100d10 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b2355700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19d740 con 0x7f71ac105410 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45680/0 (socket says 172.21.15.23:45680) 2023-11-17T00:00:01.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 -- 172.21.15.23:0/75169865 learned_addr learned my addr 172.21.15.23:0/75169865 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1b54700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71ac105410 0x7f71ac19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b0b52700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac100d10 0x7f71ac198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 -- 172.21.15.23:0/75169865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71ac105410 msgr2=0x7f71ac19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71ac105410 0x7f71ac19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 -- 172.21.15.23:0/75169865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac100d10 msgr2=0x7f71ac198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac100d10 0x7f71ac198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.901+0000 7f71b1353700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a80095f0 con 0x7f71ac100370 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71b1353700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac198450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f71a800b580 tx=0x7f71a800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a801d070 con 0x7f71ac100370 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71a8005050 con 0x7f71ac100370 2023-11-17T00:00:01.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8004a40 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac1101a0 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.902+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac110010 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.903+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71ac074070 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.905+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f71a80051c0 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.906+0000 7f71a27fc700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f71980a1260 0x7f71980a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.906+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f71a80ce060 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71b0b52700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f71980a1260 0x7f71980a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71a27fc700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] conn(0x7f71980abdd0 0x7f71980ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71980ae8d0 con 0x7f71980abdd0 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71a80ce3d0 con 0x7f71ac100370 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71b1b54700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] conn(0x7f71980abdd0 0x7f71980ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71b0b52700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f71980a1260 0x7f71980a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f71ac199a70 tx=0x7f719c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.907+0000 7f71b1b54700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] conn(0x7f71980abdd0 0x7f71980ae200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.908+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== osd.3 v2:172.21.15.23:6810/2975279435 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f71980ae8d0 con 0x7f71980abdd0 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.925+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 --> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71ac19dc80 con 0x7f71980abdd0 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.925+0000 7f71a27fc700 1 -- 172.21.15.23:0/75169865 <== osd.3 v2:172.21.15.23:6810/2975279435 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f71ac19dc80 con 0x7f71980abdd0 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] conn(0x7f71980abdd0 msgr2=0x7f71980ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] conn(0x7f71980abdd0 0x7f71980ae200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f71980a1260 msgr2=0x7f71980a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f71980a1260 0x7f71980a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f71ac199a70 tx=0x7f719c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 msgr2=0x7f71ac198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 --2- 172.21.15.23:0/75169865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac100370 0x7f71ac198450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f71a800b580 tx=0x7f71a800bfd0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b1b54700 1 -- 172.21.15.23:0/75169865 reap_dead start 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 shutdown_connections 2023-11-17T00:00:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 >> 172.21.15.23:0/75169865 conn(0x7f71ac0fbbd0 msgr2=0x7f71ac0fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:01.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 shutdown_connections 2023-11-17T00:00:01.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:00.926+0000 7f71b2355700 1 -- 172.21.15.23:0/75169865 wait complete. 2023-11-17T00:00:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2371003859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a810deb0 msgr2=0x7ff7a8115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2371003859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a810deb0 0x7ff7a8115a30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000b0a0 tx=0x7ff7a000b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2371003859 shutdown_connections 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2371003859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a810deb0 0x7ff7a8115a30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2371003859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 0x7ff7a8069790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2371003859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a8069d60 0x7ff7a810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.556+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2371003859 >> 172.21.15.23:0/2371003859 conn(0x7ff7a80719c0 msgr2=0x7ff7a8071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.557+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2371003859 shutdown_connections 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.557+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2371003859 wait complete. 2023-11-17T00:00:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.558+0000 7ff7adbf1700 1 Processor -- start 2023-11-17T00:00:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.558+0000 7ff7adbf1700 1 -- start start 2023-11-17T00:00:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 0x7ff7a8107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a8069d60 0x7ff7a8108100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a810deb0 0x7ff7a8108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81b16c0 con 0x7ff7a810deb0 2023-11-17T00:00:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81b1800 con 0x7ff7a80693c0 2023-11-17T00:00:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7adbf1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a81b1940 con 0x7ff7a8069d60 2023-11-17T00:00:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7a6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a8069d60 0x7ff7a8108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 0x7ff7a8107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a810deb0 0x7ff7a8108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7a6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a8069d60 0x7ff7a8108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:58650/0 (socket says 172.21.15.23:58650) 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.559+0000 7ff7a6ffd700 1 -- 172.21.15.23:0/2994195816 learned_addr learned my addr 172.21.15.23:0/2994195816 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 -- 172.21.15.23:0/2994195816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a8069d60 msgr2=0x7ff7a8108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7a8069d60 0x7ff7a8108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 -- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a810deb0 msgr2=0x7ff7a8108640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a810deb0 0x7ff7a8108640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a0009d00 con 0x7ff7a80693c0 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a7fff700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7a810deb0 0x7ff7a8108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a77fe700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 0x7ff7a8107bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e9f0 tx=0x7ff79800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79800c9c0 con 0x7ff7a80693c0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff798010430 con 0x7ff7a80693c0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.560+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798018680 con 0x7ff7a80693c0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.561+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a8118c10 con 0x7ff7a80693c0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.561+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a8118a80 con 0x7ff7a80693c0 2023-11-17T00:00:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.562+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7a806e220 con 0x7ff7a80693c0 2023-11-17T00:00:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.564+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff798010860 con 0x7ff7a80693c0 2023-11-17T00:00:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.565+0000 7ff7a4ff9700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff7900a11e0 0x7ff7900a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.565+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff798014070 con 0x7ff7a80693c0 2023-11-17T00:00:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a6ffd700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff7900a11e0 0x7ff7900a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a4ff9700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] conn(0x7ff7900abd50 0x7ff7900ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7900ae850 con 0x7ff7900abd50 2023-11-17T00:00:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7980ccf20 con 0x7ff7a80693c0 2023-11-17T00:00:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a7fff700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] conn(0x7ff7900abd50 0x7ff7900ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.566+0000 7ff7a6ffd700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff7900a11e0 0x7ff7900a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff7a81093b0 tx=0x7ff79c007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.567+0000 7ff7a7fff700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] conn(0x7ff7900abd50 0x7ff7900ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:01.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.568+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== osd.6 v2:172.21.15.23:6818/1429107943 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff7900ae850 con 0x7ff7900abd50 2023-11-17T00:00:01.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.590+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 --> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7a8118da0 con 0x7ff7900abd50 2023-11-17T00:00:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.590+0000 7ff7a4ff9700 1 -- 172.21.15.23:0/2994195816 <== osd.6 v2:172.21.15.23:6818/1429107943 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff7a8118da0 con 0x7ff7900abd50 2023-11-17T00:00:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] conn(0x7ff7900abd50 msgr2=0x7ff7900ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] conn(0x7ff7900abd50 0x7ff7900ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff7900a11e0 msgr2=0x7ff7900a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff7900a11e0 0x7ff7900a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff7a81093b0 tx=0x7ff79c007480 comp rx=0 tx=0).stop 2023-11-17T00:00:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 msgr2=0x7ff7a8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 --2- 172.21.15.23:0/2994195816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a80693c0 0x7ff7a8107bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e9f0 tx=0x7ff79800ed00 comp rx=0 tx=0).stop 2023-11-17T00:00:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7a7fff700 1 -- 172.21.15.23:0/2994195816 reap_dead start 2023-11-17T00:00:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 shutdown_connections 2023-11-17T00:00:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 >> 172.21.15.23:0/2994195816 conn(0x7ff7a80719c0 msgr2=0x7ff7a806d850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.591+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 shutdown_connections 2023-11-17T00:00:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:01.592+0000 7ff7adbf1700 1 -- 172.21.15.23:0/2994195816 wait complete. 2023-11-17T00:00:02.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:01 smithi023 ceph-mon[97824]: pgmap v333: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:00:02.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:01 smithi173 ceph-mon[131534]: pgmap v333: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:00:02.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:01 smithi156 ceph-mon[131105]: pgmap v333: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:00:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 -- 172.21.15.23:0/4246624750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 msgr2=0x7f899010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 --2- 172.21.15.23:0/4246624750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899010d160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f898800b0d0 tx=0x7f898800b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 -- 172.21.15.23:0/4246624750 shutdown_connections 2023-11-17T00:00:02.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 --2- 172.21.15.23:0/4246624750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899010d160 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 --2- 172.21.15.23:0/4246624750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100540 0x7f8990100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 --2- 172.21.15.23:0/4246624750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100ee0 0x7f8990104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 -- 172.21.15.23:0/4246624750 >> 172.21.15.23:0/4246624750 conn(0x7f89900fbdc0 msgr2=0x7f89900fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:02.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 -- 172.21.15.23:0/4246624750 shutdown_connections 2023-11-17T00:00:02.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.091+0000 7f8997f74700 1 -- 172.21.15.23:0/4246624750 wait complete. 2023-11-17T00:00:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.092+0000 7f8997f74700 1 Processor -- start 2023-11-17T00:00:02.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.092+0000 7f8997f74700 1 -- start start 2023-11-17T00:00:02.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100540 0x7f89901964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:02.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100ee0 0x7f89901969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:02.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019b470 con 0x7f89901055e0 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019b5e0 con 0x7f8990100540 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8997f74700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019b7a0 con 0x7f8990100ee0 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8996511700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f899550f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100ee0 0x7f89901969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8995d10700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100540 0x7f89901964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8996511700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899019ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34108/0 (socket says 172.21.15.23:34108) 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.093+0000 7f8996511700 1 -- 172.21.15.23:0/950757894 learned_addr learned my addr 172.21.15.23:0/950757894 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 -- 172.21.15.23:0/950757894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100ee0 msgr2=0x7f89901969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100ee0 0x7f89901969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 -- 172.21.15.23:0/950757894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100540 msgr2=0x7f89901964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100540 0x7f89901964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988009d00 con 0x7f89901055e0 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f899550f700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8990100ee0 0x7f89901969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8995d10700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990100540 0x7f89901964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8996511700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899019ae60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8988007820 tx=0x7f8988007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898800e040 con 0x7f89901055e0 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f898800bcb0 con 0x7f89901055e0 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89880058a0 con 0x7f89901055e0 2023-11-17T00:00:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8990110370 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.094+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89901101e0 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.095+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89900fd190 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.097+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8988004a50 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8986ffd700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f897c0a1260 0x7f897c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f89880ce500 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8995d10700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f897c0a1260 0x7f897c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8986ffd700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] conn(0x7f897c0abdd0 0x7f897c0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f897c0ae8d0 con 0x7f897c0abdd0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89880ce870 con 0x7f89901055e0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f899550f700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] conn(0x7f897c0abdd0 0x7f897c0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.099+0000 7f8995d10700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f897c0a1260 0x7f897c0a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f898c011490 tx=0x7f898c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:02.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.100+0000 7f899550f700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] conn(0x7f897c0abdd0 0x7f897c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:02.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.101+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== osd.9 v2:172.21.15.23:6826/2722703119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f897c0ae8d0 con 0x7f897c0abdd0 2023-11-17T00:00:02.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.117+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 --> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8990110500 con 0x7f897c0abdd0 2023-11-17T00:00:02.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8986ffd700 1 -- 172.21.15.23:0/950757894 <== osd.9 v2:172.21.15.23:6826/2722703119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8990110500 con 0x7f897c0abdd0 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] conn(0x7f897c0abdd0 msgr2=0x7f897c0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] conn(0x7f897c0abdd0 0x7f897c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f897c0a1260 msgr2=0x7f897c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f897c0a1260 0x7f897c0a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f898c011490 tx=0x7f898c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 msgr2=0x7f899019ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 --2- 172.21.15.23:0/950757894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89901055e0 0x7f899019ae60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8988007820 tx=0x7f8988007850 comp rx=0 tx=0).stop 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8996511700 1 -- 172.21.15.23:0/950757894 reap_dead start 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 shutdown_connections 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 >> 172.21.15.23:0/950757894 conn(0x7f89900fbdc0 msgr2=0x7f89900fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 shutdown_connections 2023-11-17T00:00:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:02.118+0000 7f8997f74700 1 -- 172.21.15.23:0/950757894 wait complete. 2023-11-17T00:00:02.880 INFO:teuthology.orchestra.run.smithi023.stdout:283467841576 2023-11-17T00:00:02.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-11-17T00:00:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/3913227890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa381055e0 msgr2=0x7ffa3810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/3913227890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa381055e0 0x7ffa3810d160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b0d0 tx=0x7ffa3000b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/3913227890 shutdown_connections 2023-11-17T00:00:03.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/3913227890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa381055e0 0x7ffa3810d160 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/3913227890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa38100540 0x7ffa38100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/3913227890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100ee0 0x7ffa38104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/3913227890 >> 172.21.15.23:0/3913227890 conn(0x7ffa380fbdc0 msgr2=0x7ffa380fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/3913227890 shutdown_connections 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.465+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/3913227890 wait complete. 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.466+0000 7ffa3e7ee700 1 Processor -- start 2023-11-17T00:00:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.466+0000 7ffa3e7ee700 1 -- start start 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100540 0x7ffa38072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 0x7ffa38073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa381055e0 0x7ffa381a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38073c20 con 0x7ffa38100ee0 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381a4c10 con 0x7ffa38100540 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3e7ee700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381a4d80 con 0x7ffa381055e0 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa377fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 0x7ffa38073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa37fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100540 0x7ffa38072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa377fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 0x7ffa38073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34128/0 (socket says 172.21.15.23:34128) 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa377fe700 1 -- 172.21.15.23:0/718078805 learned_addr learned my addr 172.21.15.23:0/718078805 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.467+0000 7ffa3cd8b700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa381055e0 0x7ffa381a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 -- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa381055e0 msgr2=0x7ffa381a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa381055e0 0x7ffa381a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 -- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 msgr2=0x7ffa38073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 0x7ffa38073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa30009d00 con 0x7ffa38100540 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa377fe700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa38100ee0 0x7ffa38073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa37fff700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100540 0x7ffa38072df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800e9f0 tx=0x7ffa2800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2800c9c0 con 0x7ffa38100540 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa38110320 con 0x7ffa38100540 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa28010430 con 0x7ffa38100540 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28018680 con 0x7ffa38100540 2023-11-17T00:00:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.468+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa38110190 con 0x7ffa38100540 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.469+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa380fd0f0 con 0x7ffa38100540 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.476+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ffa28010860 con 0x7ffa38100540 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.478+0000 7ffa357fa700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa200a1260 0x7ffa200a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.478+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa28014070 con 0x7ffa38100540 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.478+0000 7ffa377fe700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa200a1260 0x7ffa200a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa357fa700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] conn(0x7ffa200abdd0 0x7ffa200ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa200ae8d0 con 0x7ffa200abdd0 2023-11-17T00:00:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa280cced0 con 0x7ffa38100540 2023-11-17T00:00:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa3cd8b700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] conn(0x7ffa200abdd0 0x7ffa200ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa377fe700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa200a1260 0x7ffa200a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffa380745e0 tx=0x7ffa2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.479+0000 7ffa3cd8b700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] conn(0x7ffa200abdd0 0x7ffa200ae200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.480+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== osd.8 v2:172.21.15.173:6816/2703397724 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ffa200ae8d0 con 0x7ffa200abdd0 2023-11-17T00:00:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.499+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 --> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa381104b0 con 0x7ffa200abdd0 2023-11-17T00:00:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.499+0000 7ffa357fa700 1 -- 172.21.15.23:0/718078805 <== osd.8 v2:172.21.15.173:6816/2703397724 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ffa381104b0 con 0x7ffa200abdd0 2023-11-17T00:00:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] conn(0x7ffa200abdd0 msgr2=0x7ffa200ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] conn(0x7ffa200abdd0 0x7ffa200ae200 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa200a1260 msgr2=0x7ffa200a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ffa200a1260 0x7ffa200a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffa380745e0 tx=0x7ffa2c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100540 msgr2=0x7ffa38072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 --2- 172.21.15.23:0/718078805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa38100540 0x7ffa38072df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800e9f0 tx=0x7ffa2800ed00 comp rx=0 tx=0).stop 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3cd8b700 1 -- 172.21.15.23:0/718078805 reap_dead start 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 shutdown_connections 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 >> 172.21.15.23:0/718078805 conn(0x7ffa380fbdc0 msgr2=0x7ffa380fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 shutdown_connections 2023-11-17T00:00:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.500+0000 7ffa3e7ee700 1 -- 172.21.15.23:0/718078805 wait complete. 2023-11-17T00:00:03.524 INFO:teuthology.orchestra.run.smithi023.stdout:107374182469 2023-11-17T00:00:03.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 -- 172.21.15.23:0/806847072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100540 msgr2=0x7f6e5c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 --2- 172.21.15.23:0/806847072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100540 0x7f6e5c100910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009990 tx=0x7f6e4c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 -- 172.21.15.23:0/806847072 shutdown_connections 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 --2- 172.21.15.23:0/806847072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1055e0 0x7f6e5c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 --2- 172.21.15.23:0/806847072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100540 0x7f6e5c100910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 --2- 172.21.15.23:0/806847072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c100ee0 0x7f6e5c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 -- 172.21.15.23:0/806847072 >> 172.21.15.23:0/806847072 conn(0x7f6e5c0fbdc0 msgr2=0x7f6e5c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 -- 172.21.15.23:0/806847072 shutdown_connections 2023-11-17T00:00:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.542+0000 7f6e614ca700 1 -- 172.21.15.23:0/806847072 wait complete. 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.544+0000 7f6e614ca700 1 Processor -- start 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.544+0000 7f6e614ca700 1 -- start start 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 0x7f6e5c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100ee0 0x7f6e5c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1055e0 0x7f6e5c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c19b3a0 con 0x7f6e5c100540 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c19b510 con 0x7f6e5c100ee0 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e614ca700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c19b6d0 con 0x7f6e5c1055e0 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 0x7f6e5c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 0x7f6e5c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34152/0 (socket says 172.21.15.23:34152) 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e5affd700 1 -- 172.21.15.23:0/1932094059 learned_addr learned my addr 172.21.15.23:0/1932094059 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e5a7fc700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100ee0 0x7f6e5c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.545+0000 7f6e5b7fe700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1055e0 0x7f6e5c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 -- 172.21.15.23:0/1932094059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1055e0 msgr2=0x7f6e5c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1055e0 0x7f6e5c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 -- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100ee0 msgr2=0x7f6e5c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100ee0 0x7f6e5c196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0095f0 con 0x7f6e5c100540 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5b7fe700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1055e0 0x7f6e5c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5a7fc700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e5c100ee0 0x7f6e5c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e5affd700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 0x7f6e5c1963e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c005910 tx=0x7f6e4c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c01d070 con 0x7f6e5c100540 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4c022470 con 0x7f6e5c100540 2023-11-17T00:00:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c110370 con 0x7f6e5c100540 2023-11-17T00:00:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c017650 con 0x7f6e5c100540 2023-11-17T00:00:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.546+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c1101e0 con 0x7f6e5c100540 2023-11-17T00:00:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.547+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e5c0fd2c0 con 0x7f6e5c100540 2023-11-17T00:00:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.550+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e4c0225e0 con 0x7f6e5c100540 2023-11-17T00:00:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.551+0000 7f6e43fff700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e440a11e0 0x7f6e440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.551+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e4c0ceac0 con 0x7f6e5c100540 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.551+0000 7f6e5a7fc700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e440a11e0 0x7f6e440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e43fff700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] conn(0x7f6e440abd50 0x7f6e440ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e440ae850 con 0x7f6e440abd50 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e4c0cee30 con 0x7f6e5c100540 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e5b7fe700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] conn(0x7f6e440abd50 0x7f6e440ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e5a7fc700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e440a11e0 0x7f6e440a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c197a00 tx=0x7f6e50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.552+0000 7f6e5b7fe700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] conn(0x7f6e440abd50 0x7f6e440ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.553+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== osd.10 v2:172.21.15.156:6824/651591322 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6e440ae850 con 0x7f6e440abd50 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.570+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 --> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e5c110500 con 0x7f6e440abd50 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.571+0000 7f6e43fff700 1 -- 172.21.15.23:0/1932094059 <== osd.10 v2:172.21.15.156:6824/651591322 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6e5c110500 con 0x7f6e440abd50 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.571+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] conn(0x7f6e440abd50 msgr2=0x7f6e440ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.571+0000 7f6e614ca700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] conn(0x7f6e440abd50 0x7f6e440ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e440a11e0 msgr2=0x7f6e440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e440a11e0 0x7f6e440a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c197a00 tx=0x7f6e50009260 comp rx=0 tx=0).stop 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 msgr2=0x7f6e5c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 --2- 172.21.15.23:0/1932094059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c100540 0x7f6e5c1963e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c005910 tx=0x7f6e4c005720 comp rx=0 tx=0).stop 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e5b7fe700 1 -- 172.21.15.23:0/1932094059 reap_dead start 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 shutdown_connections 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 >> 172.21.15.23:0/1932094059 conn(0x7f6e5c0fbdc0 msgr2=0x7f6e5c0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 shutdown_connections 2023-11-17T00:00:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:03.572+0000 7f6e614ca700 1 -- 172.21.15.23:0/1932094059 wait complete. 2023-11-17T00:00:03.637 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:03.760 INFO:teuthology.orchestra.run.smithi023.stdout:197568495669 2023-11-17T00:00:03.760 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 -- 172.21.15.23:0/1518586286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92641055e0 msgr2=0x7f926410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 --2- 172.21.15.23:0/1518586286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92641055e0 0x7f926410d160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f925c009f90 tx=0x7f925c00b330 comp rx=0 tx=0).stop 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 -- 172.21.15.23:0/1518586286 shutdown_connections 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 --2- 172.21.15.23:0/1518586286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92641055e0 0x7f926410d160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 --2- 172.21.15.23:0/1518586286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9264100540 0x7f9264100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 --2- 172.21.15.23:0/1518586286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 0x7f9264104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 -- 172.21.15.23:0/1518586286 >> 172.21.15.23:0/1518586286 conn(0x7f92640fbdc0 msgr2=0x7f92640fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 -- 172.21.15.23:0/1518586286 shutdown_connections 2023-11-17T00:00:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.307+0000 7f926aa39700 1 -- 172.21.15.23:0/1518586286 wait complete. 2023-11-17T00:00:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.308+0000 7f926aa39700 1 Processor -- start 2023-11-17T00:00:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 -- start start 2023-11-17T00:00:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 0x7f926406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 0x7f926406b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92641055e0 0x7f92641aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641aaea0 con 0x7f9264100ee0 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641ab010 con 0x7f9264100540 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f926aa39700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641ab1d0 con 0x7f92641055e0 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f9263fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 0x7f926406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f9263fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 0x7f926406aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:56138/0 (socket says 172.21.15.23:56138) 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f9263fff700 1 -- 172.21.15.23:0/864959667 learned_addr learned my addr 172.21.15.23:0/864959667 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:04.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.309+0000 7f92637fe700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 0x7f926406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9268fd6700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92641055e0 0x7f92641aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 -- 172.21.15.23:0/864959667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92641055e0 msgr2=0x7f92641aa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92641055e0 0x7f92641aa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 -- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 msgr2=0x7f926406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 0x7f926406b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580095f0 con 0x7f9264100540 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f92637fe700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9264100ee0 0x7f926406b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f9263fff700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 0x7f926406aff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f925400e9c0 tx=0x7f925400ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92540097b0 con 0x7f9264100540 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9254004cf0 con 0x7f9264100540 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254005700 con 0x7f9264100540 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.310+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c009c40 con 0x7f9264100540 2023-11-17T00:00:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.311+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9264110340 con 0x7f9264100540 2023-11-17T00:00:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.312+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9250000ff0 con 0x7f9264100540 2023-11-17T00:00:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.314+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9254004830 con 0x7f9264100540 2023-11-17T00:00:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.315+0000 7f92617fa700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f924c0a1260 0x7f924c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.315+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9254014070 con 0x7f9264100540 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f92617fa700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] conn(0x7f924c0abdd0 0x7f924c0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f924c0ae8d0 con 0x7f924c0abdd0 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92540cf420 con 0x7f9264100540 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f92637fe700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f924c0a1260 0x7f924c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f9268fd6700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] conn(0x7f924c0abdd0 0x7f924c0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.316+0000 7f92637fe700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f924c0a1260 0x7f924c0a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f926406c670 tx=0x7f9258009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.317+0000 7f9268fd6700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] conn(0x7f924c0abdd0 0x7f924c0ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.318+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== osd.1 v2:172.21.15.156:6800/2641693166 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f924c0ae8d0 con 0x7f924c0abdd0 2023-11-17T00:00:04.322 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:04 smithi023 ceph-mon[97824]: pgmap v334: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-17T00:00:04.322 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:04 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.347+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 --> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9250002d70 con 0x7f924c0abdd0 2023-11-17T00:00:04.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.347+0000 7f92617fa700 1 -- 172.21.15.23:0/864959667 <== osd.1 v2:172.21.15.156:6800/2641693166 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9250002d70 con 0x7f924c0abdd0 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] conn(0x7f924c0abdd0 msgr2=0x7f924c0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] conn(0x7f924c0abdd0 0x7f924c0ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f924c0a1260 msgr2=0x7f924c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f924c0a1260 0x7f924c0a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f926406c670 tx=0x7f9258009310 comp rx=0 tx=0).stop 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 msgr2=0x7f926406aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f926aa39700 1 --2- 172.21.15.23:0/864959667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9264100540 0x7f926406aff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f925400e9c0 tx=0x7f925400ed80 comp rx=0 tx=0).stop 2023-11-17T00:00:04.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.348+0000 7f9268fd6700 1 -- 172.21.15.23:0/864959667 reap_dead start 2023-11-17T00:00:04.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.350+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 shutdown_connections 2023-11-17T00:00:04.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.350+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 >> 172.21.15.23:0/864959667 conn(0x7f92640fbdc0 msgr2=0x7f92640fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.350+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 shutdown_connections 2023-11-17T00:00:04.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.350+0000 7f926aa39700 1 -- 172.21.15.23:0/864959667 wait complete. 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 -- 172.21.15.23:0/284013089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c100540 msgr2=0x7f4c8c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 --2- 172.21.15.23:0/284013089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c100540 0x7f4c8c100910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009990 tx=0x7f4c88009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 -- 172.21.15.23:0/284013089 shutdown_connections 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 --2- 172.21.15.23:0/284013089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c1055e0 0x7f4c8c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 --2- 172.21.15.23:0/284013089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c100540 0x7f4c8c100910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 --2- 172.21.15.23:0/284013089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 -- 172.21.15.23:0/284013089 >> 172.21.15.23:0/284013089 conn(0x7f4c8c0fbdc0 msgr2=0x7f4c8c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.560+0000 7f4c93db5700 1 -- 172.21.15.23:0/284013089 shutdown_connections 2023-11-17T00:00:04.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.561+0000 7f4c93db5700 1 -- 172.21.15.23:0/284013089 wait complete. 2023-11-17T00:00:04.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 Processor -- start 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 -- start start 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c8c100540 0x7f4c8c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c1055e0 0x7f4c8c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c19d5c0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c19d730 con 0x7f4c8c1055e0 2023-11-17T00:00:04.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c93db5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c19d8f0 con 0x7f4c8c100540 2023-11-17T00:00:04.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c91350700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.562+0000 7f4c91b51700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c8c100540 0x7f4c8c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34182/0 (socket says 172.21.15.23:34182) 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 -- 172.21.15.23:0/140442050 learned_addr learned my addr 172.21.15.23:0/140442050 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c92352700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c1055e0 0x7f4c8c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 -- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c8c100540 msgr2=0x7f4c8c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c8c100540 0x7f4c8c198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 -- 172.21.15.23:0/140442050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c1055e0 msgr2=0x7f4c8c19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c8c1055e0 0x7f4c8c19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880095f0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c91350700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c198b40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00e970 tx=0x7f4c7c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c009790 con 0x7f4c8c100ee0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c7c00c7e0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.563+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c110370 con 0x7f4c8c100ee0 2023-11-17T00:00:04.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.564+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c1101e0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.564+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0054e0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.565+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c70000ff0 con 0x7f4c8c100ee0 2023-11-17T00:00:04.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.567+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4c7c004a30 con 0x7f4c8c100ee0 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c82ffd700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c780a1260 0x7f4c780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4c7c014070 con 0x7f4c8c100ee0 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c91b51700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c780a1260 0x7f4c780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c82ffd700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] conn(0x7f4c780abdd0 0x7f4c780ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c780ae8d0 con 0x7f4c780abdd0 2023-11-17T00:00:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.568+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c7c0cd610 con 0x7f4c8c100ee0 2023-11-17T00:00:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.569+0000 7f4c92352700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] conn(0x7f4c780abdd0 0x7f4c780ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.569+0000 7f4c91b51700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c780a1260 0x7f4c780a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800b2e0 tx=0x7f4c88005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.569+0000 7f4c92352700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] conn(0x7f4c780abdd0 0x7f4c780ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.570+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== osd.5 v2:172.21.15.173:6808/2003227245 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4c780ae8d0 con 0x7f4c780abdd0 2023-11-17T00:00:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.587+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 --> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c70002d70 con 0x7f4c780abdd0 2023-11-17T00:00:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c82ffd700 1 -- 172.21.15.23:0/140442050 <== osd.5 v2:172.21.15.173:6808/2003227245 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4c70002d70 con 0x7f4c780abdd0 2023-11-17T00:00:04.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] conn(0x7f4c780abdd0 msgr2=0x7f4c780ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] conn(0x7f4c780abdd0 0x7f4c780ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c780a1260 msgr2=0x7f4c780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c780a1260 0x7f4c780a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800b2e0 tx=0x7f4c88005fb0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 msgr2=0x7f4c8c198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c93db5700 1 --2- 172.21.15.23:0/140442050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8c100ee0 0x7f4c8c198b40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00e970 tx=0x7f4c7c00ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.588+0000 7f4c92352700 1 -- 172.21.15.23:0/140442050 reap_dead start 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.589+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 shutdown_connections 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.589+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 >> 172.21.15.23:0/140442050 conn(0x7f4c8c0fbdc0 msgr2=0x7f4c8c0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.589+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 shutdown_connections 2023-11-17T00:00:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.589+0000 7f4c93db5700 1 -- 172.21.15.23:0/140442050 wait complete. 2023-11-17T00:00:04.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:04 smithi156 ceph-mon[131105]: pgmap v334: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-17T00:00:04.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.634 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:04 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:04 smithi173 ceph-mon[131534]: pgmap v334: 129 pgs: 129 active+clean; 638 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-11-17T00:00:04.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:04 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:04.740 INFO:teuthology.orchestra.run.smithi023.stdout:304942678047 2023-11-17T00:00:04.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-11-17T00:00:04.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 -- 172.21.15.23:0/3983173709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a00ffc40 msgr2=0x7f72a0104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/3983173709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a00ffc40 0x7f72a0104120 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7294009990 tx=0x7f7294009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 -- 172.21.15.23:0/3983173709 shutdown_connections 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/3983173709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0104660 0x7f72a0104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/3983173709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ff330 0x7f72a00ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/3983173709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a00ffc40 0x7f72a0104120 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.895+0000 7f72a5bcd700 1 -- 172.21.15.23:0/3983173709 >> 172.21.15.23:0/3983173709 conn(0x7f72a00770a0 msgr2=0x7f72a00774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.896+0000 7f72a5bcd700 1 -- 172.21.15.23:0/3983173709 shutdown_connections 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.896+0000 7f72a5bcd700 1 -- 172.21.15.23:0/3983173709 wait complete. 2023-11-17T00:00:04.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.897+0000 7f72a5bcd700 1 Processor -- start 2023-11-17T00:00:04.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.897+0000 7f72a5bcd700 1 -- start start 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 0x7f72a01a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ffc40 0x7f72a01a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a0104660 0x7f72a019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019cfe0 con 0x7f72a0104660 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019d150 con 0x7f72a00ff330 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f72a5bcd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019d310 con 0x7f72a00ffc40 2023-11-17T00:00:04.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 0x7f72a01a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 0x7f72a01a28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:56176/0 (socket says 172.21.15.23:56176) 2023-11-17T00:00:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.898+0000 7f729f7fe700 1 -- 172.21.15.23:0/337459197 learned_addr learned my addr 172.21.15.23:0/337459197 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729effd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ffc40 0x7f72a01a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729ffff700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a0104660 0x7f72a019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729f7fe700 1 -- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ffc40 msgr2=0x7f72a01a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729f7fe700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ffc40 0x7f72a01a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729f7fe700 1 -- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a0104660 msgr2=0x7f72a019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729f7fe700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72a0104660 0x7f72a019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729f7fe700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72940095f0 con 0x7f72a00ff330 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.899+0000 7f729effd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a00ffc40 0x7f72a01a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f729f7fe700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 0x7f72a01a28f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f729000b550 tx=0x7f729000b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7290011830 con 0x7f72a00ff330 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7290011e50 con 0x7f72a00ff330 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a01169f0 con 0x7f72a00ff330 2023-11-17T00:00:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7290010e40 con 0x7f72a00ff330 2023-11-17T00:00:04.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.900+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a0116860 con 0x7f72a00ff330 2023-11-17T00:00:04.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.901+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72a0103b90 con 0x7f72a00ff330 2023-11-17T00:00:04.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.903+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7290010380 con 0x7f72a00ff330 2023-11-17T00:00:04.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.904+0000 7f729cff9700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72880a1260 0x7f72880a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.905+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f72900cc5a0 con 0x7f72a00ff330 2023-11-17T00:00:04.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.905+0000 7f729effd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72880a1260 0x7f72880a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.905+0000 7f729cff9700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] conn(0x7f72880abdd0 0x7f72880ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.905+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72880ae8d0 con 0x7f72880abdd0 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.905+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72900ceb80 con 0x7f72a00ff330 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.906+0000 7f729ffff700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] conn(0x7f72880abdd0 0x7f72880ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.906+0000 7f729effd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72880a1260 0x7f72880a3710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f72940098e0 tx=0x7f7294005ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.906+0000 7f729ffff700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] conn(0x7f72880abdd0 0x7f72880ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:04.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.907+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== osd.2 v2:172.21.15.173:6800/2331881016 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f72880ae8d0 con 0x7f72880abdd0 2023-11-17T00:00:04.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.924+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 --> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f72a0116b80 con 0x7f72880abdd0 2023-11-17T00:00:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.924+0000 7f729cff9700 1 -- 172.21.15.23:0/337459197 <== osd.2 v2:172.21.15.173:6800/2331881016 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f72a0116b80 con 0x7f72880abdd0 2023-11-17T00:00:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] conn(0x7f72880abdd0 msgr2=0x7f72880ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] conn(0x7f72880abdd0 0x7f72880ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72880a1260 msgr2=0x7f72880a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72880a1260 0x7f72880a3710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f72940098e0 tx=0x7f7294005ab0 comp rx=0 tx=0).stop 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 msgr2=0x7f72a01a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 --2- 172.21.15.23:0/337459197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a00ff330 0x7f72a01a28f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f729000b550 tx=0x7f729000b860 comp rx=0 tx=0).stop 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f729ffff700 1 -- 172.21.15.23:0/337459197 reap_dead start 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 shutdown_connections 2023-11-17T00:00:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 >> 172.21.15.23:0/337459197 conn(0x7f72a00770a0 msgr2=0x7f72a01021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 shutdown_connections 2023-11-17T00:00:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:04.925+0000 7f72a5bcd700 1 -- 172.21.15.23:0/337459197 wait complete. 2023-11-17T00:00:05.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 -- 172.21.15.23:0/1246755641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a30101540 msgr2=0x7f0a30101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 --2- 172.21.15.23:0/1246755641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a30101540 0x7f0a30101910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0a200098f0 tx=0x7f0a20009c00 comp rx=0 tx=0).stop 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 -- 172.21.15.23:0/1246755641 shutdown_connections 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 --2- 172.21.15.23:0/1246755641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3010a890 0x7f0a3010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 --2- 172.21.15.23:0/1246755641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a30101540 0x7f0a30101910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 --2- 172.21.15.23:0/1246755641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 0x7f0a3010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 -- 172.21.15.23:0/1246755641 >> 172.21.15.23:0/1246755641 conn(0x7f0a300770a0 msgr2=0x7f0a300774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 -- 172.21.15.23:0/1246755641 shutdown_connections 2023-11-17T00:00:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.300+0000 7f0a365e8700 1 -- 172.21.15.23:0/1246755641 wait complete. 2023-11-17T00:00:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.302+0000 7f0a365e8700 1 Processor -- start 2023-11-17T00:00:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.302+0000 7f0a365e8700 1 -- start start 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.302+0000 7f0a365e8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a30101540 0x7f0a300ff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.302+0000 7f0a365e8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 0x7f0a300ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a365e8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3010a890 0x7f0a300ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a365e8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a301a4960 con 0x7f0a30101e50 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a365e8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a301a4ad0 con 0x7f0a30101540 2023-11-17T00:00:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a365e8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a301a4c90 con 0x7f0a3010a890 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a30101540 0x7f0a300ff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 0x7f0a300ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 0x7f0a300ff6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34228/0 (socket says 172.21.15.23:34228) 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2f7fe700 1 -- 172.21.15.23:0/2697101690 learned_addr learned my addr 172.21.15.23:0/2697101690 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a34b85700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3010a890 0x7f0a300ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 -- 172.21.15.23:0/2697101690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3010a890 msgr2=0x7f0a300ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3010a890 0x7f0a300ffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 -- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 msgr2=0x7f0a300ff6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a30101e50 0x7f0a300ff6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.303+0000 7f0a2ffff700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20009520 con 0x7f0a30101540 2023-11-17T00:00:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a30101540 0x7f0a300ff1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009810 tx=0x7f0a2000fa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2001c070 con 0x7f0a30101540 2023-11-17T00:00:05.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a30110360 con 0x7f0a30101540 2023-11-17T00:00:05.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a2000bba0 con 0x7f0a30101540 2023-11-17T00:00:05.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20003970 con 0x7f0a30101540 2023-11-17T00:00:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.304+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301101d0 con 0x7f0a30101540 2023-11-17T00:00:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.305+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a300fd860 con 0x7f0a30101540 2023-11-17T00:00:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.307+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0a20020030 con 0x7f0a30101540 2023-11-17T00:00:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2d7fa700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0a180a11e0 0x7f0a180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0a200cdae0 con 0x7f0a30101540 2023-11-17T00:00:05.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0a180a11e0 0x7f0a180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2d7fa700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] conn(0x7f0a180abd50 0x7f0a180ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a180ae850 con 0x7f0a180abd50 2023-11-17T00:00:05.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.309+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a200d22a0 con 0x7f0a30101540 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.310+0000 7f0a34b85700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] conn(0x7f0a180abd50 0x7f0a180ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.310+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0a180a11e0 0x7f0a180a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a30100990 tx=0x7f0a24009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.310+0000 7f0a34b85700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] conn(0x7f0a180abd50 0x7f0a180ae180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.311+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== osd.7 v2:172.21.15.156:6816/2873383539 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0a180ae850 con 0x7f0a180abd50 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 -- 172.21.15.23:0/2658072164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 msgr2=0x7f6310104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 --2- 172.21.15.23:0/2658072164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 0x7f6310104eb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f63040099c0 tx=0x7f6304009cd0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 -- 172.21.15.23:0/2658072164 shutdown_connections 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 --2- 172.21.15.23:0/2658072164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63101055e0 0x7f631010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 --2- 172.21.15.23:0/2658072164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6310100540 0x7f6310100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 --2- 172.21.15.23:0/2658072164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 0x7f6310104eb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 -- 172.21.15.23:0/2658072164 >> 172.21.15.23:0/2658072164 conn(0x7f63100fbdc0 msgr2=0x7f63100fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 -- 172.21.15.23:0/2658072164 shutdown_connections 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 --> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a301104f0 con 0x7f0a180abd50 2023-11-17T00:00:05.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.345+0000 7f6315f74700 1 -- 172.21.15.23:0/2658072164 wait complete. 2023-11-17T00:00:05.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.346+0000 7f0a2d7fa700 1 -- 172.21.15.23:0/2697101690 <== osd.7 v2:172.21.15.156:6816/2873383539 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0a301104f0 con 0x7f0a180abd50 2023-11-17T00:00:05.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 Processor -- start 2023-11-17T00:00:05.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] conn(0x7f0a180abd50 msgr2=0x7f0a180ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] conn(0x7f0a180abd50 0x7f0a180ae180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 -- start start 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0a180a11e0 msgr2=0x7f0a180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0a180a11e0 0x7f0a180a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a30100990 tx=0x7f0a24009310 comp rx=0 tx=0).stop 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a30101540 msgr2=0x7f0a300ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 --2- 172.21.15.23:0/2697101690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a30101540 0x7f0a300ff1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009810 tx=0x7f0a2000fa50 comp rx=0 tx=0).stop 2023-11-17T00:00:05.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 0x7f6310198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 0x7f6310198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a34b85700 1 -- 172.21.15.23:0/2697101690 reap_dead start 2023-11-17T00:00:05.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 shutdown_connections 2023-11-17T00:00:05.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 >> 172.21.15.23:0/2697101690 conn(0x7f0a300770a0 msgr2=0x7f0a300fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:05.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 shutdown_connections 2023-11-17T00:00:05.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f0a365e8700 1 -- 172.21.15.23:0/2697101690 wait complete. 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63101055e0 0x7f631019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019d530 con 0x7f6310100540 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019d6a0 con 0x7f63101055e0 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.347+0000 7f6315f74700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019d860 con 0x7f6310100ee0 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 0x7f6310198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 0x7f6310198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34254/0 (socket says 172.21.15.23:34254) 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630f7fe700 1 -- 172.21.15.23:0/1427986091 learned_addr learned my addr 172.21.15.23:0/1427986091 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63101055e0 0x7f631019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630effd700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 0x7f6310198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 -- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 msgr2=0x7f6310198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6310100ee0 0x7f6310198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 -- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 msgr2=0x7f6310198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 0x7f6310198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630ffff700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040095f0 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.348+0000 7f630f7fe700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6310100540 0x7f6310198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f630ffff700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63101055e0 0x7f631019cf20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f630800efc0 tx=0x7f630800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308018070 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f630800f040 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6310110310 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630800cd40 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.349+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6310110180 con 0x7f63101055e0 2023-11-17T00:00:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.350+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6310073ff0 con 0x7f63101055e0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.352+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6308003a00 con 0x7f63101055e0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630cff9700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f62f80a1260 0x7f62f80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f63080cc8a0 con 0x7f63101055e0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630f7fe700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f62f80a1260 0x7f62f80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630cff9700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] conn(0x7f62f80abdd0 0x7f62f80ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62f80ae8d0 con 0x7f62f80abdd0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.354+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63080ccc10 con 0x7f63101055e0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.355+0000 7f630effd700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] conn(0x7f62f80abdd0 0x7f62f80ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.355+0000 7f630f7fe700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f62f80a1260 0x7f62f80a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6300011550 tx=0x7f6300009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.355+0000 7f630effd700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] conn(0x7f62f80abdd0 0x7f62f80ae200 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.356+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== osd.11 v2:172.21.15.173:6824/3375444046 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f62f80ae8d0 con 0x7f62f80abdd0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.374+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 --> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f63101104a0 con 0x7f62f80abdd0 2023-11-17T00:00:05.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.374+0000 7f630cff9700 1 -- 172.21.15.23:0/1427986091 <== osd.11 v2:172.21.15.173:6824/3375444046 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f63101104a0 con 0x7f62f80abdd0 2023-11-17T00:00:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] conn(0x7f62f80abdd0 msgr2=0x7f62f80ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] conn(0x7f62f80abdd0 0x7f62f80ae200 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f62f80a1260 msgr2=0x7f62f80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f62f80a1260 0x7f62f80a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6300011550 tx=0x7f6300009260 comp rx=0 tx=0).stop 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63101055e0 msgr2=0x7f631019cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 --2- 172.21.15.23:0/1427986091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63101055e0 0x7f631019cf20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f630800efc0 tx=0x7f630800c3a0 comp rx=0 tx=0).stop 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f630ffff700 1 -- 172.21.15.23:0/1427986091 reap_dead start 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 shutdown_connections 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.375+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 >> 172.21.15.23:0/1427986091 conn(0x7f63100fbdc0 msgr2=0x7f63100fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.376+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 shutdown_connections 2023-11-17T00:00:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:05.376+0000 7f6315f74700 1 -- 172.21.15.23:0/1427986091 wait complete. 2023-11-17T00:00:05.812 INFO:teuthology.orchestra.run.smithi023.stdout:249108103212 2023-11-17T00:00:05.813 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-11-17T00:00:06.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:06 smithi023 ceph-mon[97824]: pgmap v335: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-17T00:00:06.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:00:06.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:00:06.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:06 smithi156 ceph-mon[131105]: pgmap v335: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:06 smithi173 ceph-mon[131534]: pgmap v335: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-17T00:00:06.779 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:00:06.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:00:06.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:00:06.780 INFO:teuthology.orchestra.run.smithi023.stdout:163208757308 2023-11-17T00:00:06.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-17T00:00:06.782 INFO:teuthology.orchestra.run.smithi023.stdout:339302416411 2023-11-17T00:00:06.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-11-17T00:00:06.783 INFO:teuthology.orchestra.run.smithi023.stdout:77309411403 2023-11-17T00:00:06.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-17T00:00:06.839 INFO:teuthology.orchestra.run.smithi023.stdout:219043332144 2023-11-17T00:00:06.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-17T00:00:07.011 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:07.205 INFO:teuthology.orchestra.run.smithi023.stdout:55834574928 2023-11-17T00:00:07.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-17T00:00:07.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:07 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:07 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:00:07.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:07 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:07 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:00:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:07.575 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:07.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:07 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:07 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:07 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:07 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:07 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:07 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:07 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:00:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:07 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.703+0000 7f8bd17bd700 1 -- 172.21.15.23:0/307754941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc10a890 msgr2=0x7f8bcc10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.703+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/307754941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc10a890 0x7f8bcc10ccd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400b0a0 tx=0x7f8bc400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 -- 172.21.15.23:0/307754941 shutdown_connections 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/307754941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc10a890 0x7f8bcc10ccd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/307754941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc101540 0x7f8bcc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/307754941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101e50 0x7f8bcc10a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 -- 172.21.15.23:0/307754941 >> 172.21.15.23:0/307754941 conn(0x7f8bcc0770a0 msgr2=0x7f8bcc0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 -- 172.21.15.23:0/307754941 shutdown_connections 2023-11-17T00:00:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.704+0000 7f8bd17bd700 1 -- 172.21.15.23:0/307754941 wait complete. 2023-11-17T00:00:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.705+0000 7f8bd17bd700 1 Processor -- start 2023-11-17T00:00:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.705+0000 7f8bd17bd700 1 -- start start 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 0x7f8bcc196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc10a890 0x7f8bcc19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc19b310 con 0x7f8bcc101e50 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc19b480 con 0x7f8bcc101540 2023-11-17T00:00:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.706+0000 7f8bd17bd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bcc19b640 con 0x7f8bcc10a890 2023-11-17T00:00:07.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bca7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 0x7f8bcc196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:07.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bcaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:07.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bcaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:56248/0 (socket says 172.21.15.23:56248) 2023-11-17T00:00:07.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bcaffd700 1 -- 172.21.15.23:0/2701048685 learned_addr learned my addr 172.21.15.23:0/2701048685 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bca7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 0x7f8bcc196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34274/0 (socket says 172.21.15.23:34274) 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.707+0000 7f8bcb7fe700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc10a890 0x7f8bcc19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.708+0000 7f8bcaffd700 1 -- 172.21.15.23:0/2701048685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc10a890 msgr2=0x7f8bcc19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.708+0000 7f8bcaffd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc10a890 0x7f8bcc19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.708+0000 7f8bcaffd700 1 -- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 msgr2=0x7f8bcc196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.708+0000 7f8bcaffd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 0x7f8bcc196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.708+0000 7f8bcaffd700 1 -- 172.21.15.23:0/2701048685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc4009d00 con 0x7f8bcc101540 2023-11-17T00:00:07.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.710+0000 7f8bcaffd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00e9a0 tx=0x7f8bbc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:07.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.710+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00ca10 con 0x7f8bcc101540 2023-11-17T00:00:07.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.710+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bbc00cb70 con 0x7f8bcc101540 2023-11-17T00:00:07.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.710+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc018680 con 0x7f8bcc101540 2023-11-17T00:00:07.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.712+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc110350 con 0x7f8bcc101540 2023-11-17T00:00:07.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.712+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc1101c0 con 0x7f8bcc101540 2023-11-17T00:00:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.713+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc1104e0 con 0x7f8bcc101540 2023-11-17T00:00:07.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.717+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8bbc00cce0 con 0x7f8bcc101540 2023-11-17T00:00:07.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.720+0000 7f8bb3fff700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 0x7f8bb40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:07.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.720+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8bbc014070 con 0x7f8bcc101540 2023-11-17T00:00:07.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.720+0000 7f8bca7fc700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 0x7f8bb40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.721+0000 7f8bca7fc700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 0x7f8bb40a37d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc197970 tx=0x7f8bc0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.723+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8bbc096920 con 0x7f8bcc101540 2023-11-17T00:00:07.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.986+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8bcc19b890 con 0x7f8bcc101540 2023-11-17T00:00:07.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.987+0000 7f8bb3fff700 1 -- 172.21.15.23:0/2701048685 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8bbc016330 con 0x7f8bcc101540 2023-11-17T00:00:07.988 INFO:teuthology.orchestra.run.smithi023.stdout:34359738453 2023-11-17T00:00:07.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.989+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 msgr2=0x7f8bb40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:07.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.989+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 0x7f8bb40a37d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc197970 tx=0x7f8bc0009260 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.989+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 msgr2=0x7f8bcc196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.989+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00e9a0 tx=0x7f8bbc00ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 shutdown_connections 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8bb40a1320 0x7f8bb40a37d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bcc10a890 0x7f8bcc19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bcc101540 0x7f8bcc196350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 --2- 172.21.15.23:0/2701048685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bcc101e50 0x7f8bcc196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 >> 172.21.15.23:0/2701048685 conn(0x7f8bcc0770a0 msgr2=0x7f8bcc1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 shutdown_connections 2023-11-17T00:00:07.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:07.990+0000 7f8bd17bd700 1 -- 172.21.15.23:0/2701048685 wait complete. 2023-11-17T00:00:08.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:08 smithi023 ceph-mon[97824]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:08.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:08 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2701048685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T00:00:08.282 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:08.348 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:08.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:08 smithi173 ceph-mon[131534]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:08 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2701048685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T00:00:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:08 smithi156 ceph-mon[131105]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:08 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2701048685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T00:00:08.978 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738453 for osd.0 2023-11-17T00:00:09.005 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:09.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:09 smithi023 ceph-mon[97824]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:10.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:09 smithi156 ceph-mon[131105]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:10.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:09 smithi173 ceph-mon[131534]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:11.016 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:11.074 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.206+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3112098426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e01055e0 msgr2=0x7fa5e010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.206+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3112098426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e01055e0 0x7fa5e010d160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800b0d0 tx=0x7fa5d800b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3112098426 shutdown_connections 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3112098426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e01055e0 0x7fa5e010d160 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3112098426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3112098426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e0100ee0 0x7fa5e0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3112098426 >> 172.21.15.23:0/3112098426 conn(0x7fa5e00fbdc0 msgr2=0x7fa5e00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3112098426 shutdown_connections 2023-11-17T00:00:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.207+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3112098426 wait complete. 2023-11-17T00:00:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 Processor -- start 2023-11-17T00:00:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 -- start start 2023-11-17T00:00:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e0100ee0 0x7fa5e01969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019b420 con 0x7fa5e01055e0 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019b590 con 0x7fa5e0100540 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.208+0000 7fa5e62ee700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019b750 con 0x7fa5e0100ee0 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5dffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53426/0 (socket says 172.21.15.23:53426) 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5dffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:51562/0 (socket says 172.21.15.23:51562) 2023-11-17T00:00:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 -- 172.21.15.23:0/3320666731 learned_addr learned my addr 172.21.15.23:0/3320666731 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5df7fe700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e0100ee0 0x7fa5e01969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 -- 172.21.15.23:0/3320666731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e0100ee0 msgr2=0x7fa5e01969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e0100ee0 0x7fa5e01969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 -- 172.21.15.23:0/3320666731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 msgr2=0x7fa5e0196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 -- 172.21.15.23:0/3320666731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4009520 con 0x7fa5e01055e0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.209+0000 7fa5e488b700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007870 tx=0x7fa5d80078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.210+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d800e040 con 0x7fa5e01055e0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.210+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d8009d00 con 0x7fa5e01055e0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.210+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5d8005070 con 0x7fa5e01055e0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.210+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d801b6c0 con 0x7fa5e01055e0 2023-11-17T00:00:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.210+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e0110320 con 0x7fa5e01055e0 2023-11-17T00:00:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.211+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e0110190 con 0x7fa5e01055e0 2023-11-17T00:00:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.214+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa5d80051e0 con 0x7fa5e01055e0 2023-11-17T00:00:11.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.215+0000 7fa5dd7fa700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 0x7fa5c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.215+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa5d80cdd70 con 0x7fa5e01055e0 2023-11-17T00:00:11.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.215+0000 7fa5dffff700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 0x7fa5c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.216+0000 7fa5dffff700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 0x7fa5c80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009c60 tx=0x7fa5d0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:11.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.219+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa5d80971a0 con 0x7fa5e01055e0 2023-11-17T00:00:11.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.472+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa5e01104b0 con 0x7fa5e01055e0 2023-11-17T00:00:11.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.473+0000 7fa5dd7fa700 1 -- 172.21.15.23:0/3320666731 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa5d809a7c0 con 0x7fa5e01055e0 2023-11-17T00:00:11.474 INFO:teuthology.orchestra.run.smithi023.stdout:141733920832 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 msgr2=0x7fa5c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 0x7fa5c80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009c60 tx=0x7fa5d0009260 comp rx=0 tx=0).stop 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 msgr2=0x7fa5e019ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007870 tx=0x7fa5d80078a0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 shutdown_connections 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5e01055e0 0x7fa5e019ae10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5e0100540 0x7fa5e0196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa5c80a11e0 0x7fa5c80a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 --2- 172.21.15.23:0/3320666731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5e0100ee0 0x7fa5e01969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 >> 172.21.15.23:0/3320666731 conn(0x7fa5e00fbdc0 msgr2=0x7fa5e00fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:11.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 shutdown_connections 2023-11-17T00:00:11.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.475+0000 7fa5e62ee700 1 -- 172.21.15.23:0/3320666731 wait complete. 2023-11-17T00:00:11.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 -- 172.21.15.23:0/2969431692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c101e50 msgr2=0x7f892c10a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 --2- 172.21.15.23:0/2969431692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c101e50 0x7f892c10a350 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8918009990 tx=0x7f8918009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 -- 172.21.15.23:0/2969431692 shutdown_connections 2023-11-17T00:00:11.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 --2- 172.21.15.23:0/2969431692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c10a890 0x7f892c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 --2- 172.21.15.23:0/2969431692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 --2- 172.21.15.23:0/2969431692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c101e50 0x7f892c10a350 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.493+0000 7f8932a76700 1 -- 172.21.15.23:0/2969431692 >> 172.21.15.23:0/2969431692 conn(0x7f892c0770a0 msgr2=0x7f892c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:11.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.494+0000 7f8932a76700 1 -- 172.21.15.23:0/2969431692 shutdown_connections 2023-11-17T00:00:11.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.494+0000 7f8932a76700 1 -- 172.21.15.23:0/2969431692 wait complete. 2023-11-17T00:00:11.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 Processor -- start 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 -- start start 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 0x7f892c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 0x7f892c19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c19c5d0 con 0x7f892c10a890 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c19c740 con 0x7f892c101e50 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.495+0000 7f8932a76700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c19c900 con 0x7f892c101540 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 0x7f892c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f8931013700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 0x7f892c19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f8930812700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f8930812700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:35448/0 (socket says 172.21.15.23:35448) 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f8931013700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 0x7f892c19bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53454/0 (socket says 172.21.15.23:53454) 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f8930812700 1 -- 172.21.15.23:0/1699364295 learned_addr learned my addr 172.21.15.23:0/1699364295 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 -- 172.21.15.23:0/1699364295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 msgr2=0x7f892c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 -- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 msgr2=0x7f892c19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 0x7f892c19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 -- 172.21.15.23:0/1699364295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180095f0 con 0x7f892c101e50 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.496+0000 7f892bfff700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 0x7f892c198c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89180098e0 tx=0x7f8918005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.497+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891801d070 con 0x7f892c101e50 2023-11-17T00:00:11.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.497+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8918021410 con 0x7f892c101e50 2023-11-17T00:00:11.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.497+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c110360 con 0x7f892c101e50 2023-11-17T00:00:11.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.497+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918017650 con 0x7f892c101e50 2023-11-17T00:00:11.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.497+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c1101d0 con 0x7f892c101e50 2023-11-17T00:00:11.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.498+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c1104f0 con 0x7f892c101e50 2023-11-17T00:00:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.500+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8918021580 con 0x7f892c101e50 2023-11-17T00:00:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.502+0000 7f8929ffb700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 0x7f891c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.502+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f89180cdf90 con 0x7f892c101e50 2023-11-17T00:00:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.502+0000 7f8930812700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 0x7f891c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.503+0000 7f8930812700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 0x7f891c0a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8920009d20 tx=0x7f8920009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:11.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.504+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f89180974b0 con 0x7f892c101e50 2023-11-17T00:00:11.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.725+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f892c066ea0 con 0x7f892c101e50 2023-11-17T00:00:11.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.726+0000 7f8929ffb700 1 -- 172.21.15.23:0/1699364295 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f891809aad0 con 0x7f892c101e50 2023-11-17T00:00:11.727 INFO:teuthology.orchestra.run.smithi023.stdout:283467841577 2023-11-17T00:00:11.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.728+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 msgr2=0x7f891c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.728+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 0x7f891c0a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8920009d20 tx=0x7f8920009260 comp rx=0 tx=0).stop 2023-11-17T00:00:11.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.728+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 msgr2=0x7f892c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:11.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.728+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 0x7f892c198c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89180098e0 tx=0x7f8918005720 comp rx=0 tx=0).stop 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 shutdown_connections 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f892c10a890 0x7f892c19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f891c0a11e0 0x7f891c0a3690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f892c101540 0x7f892c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 --2- 172.21.15.23:0/1699364295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892c101e50 0x7f892c198c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 >> 172.21.15.23:0/1699364295 conn(0x7f892c0770a0 msgr2=0x7f892c100230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 shutdown_connections 2023-11-17T00:00:11.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:11.729+0000 7f8932a76700 1 -- 172.21.15.23:0/1699364295 wait complete. 2023-11-17T00:00:12.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:12 smithi023 ceph-mon[97824]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:00:12.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:12 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3320666731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T00:00:12.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:12 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1699364295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-17T00:00:12.521 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841576 got 283467841577 for osd.9 2023-11-17T00:00:12.521 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:12.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:12 smithi173 ceph-mon[131534]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:00:12.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:12 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3320666731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T00:00:12.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:12 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1699364295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-17T00:00:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:12 smithi156 ceph-mon[131105]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:00:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:12 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3320666731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T00:00:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:12 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1699364295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/2965700635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 msgr2=0x7f208c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- 172.21.15.23:0/2965700635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c101910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20880098f0 tx=0x7f2088009c00 comp rx=0 tx=0).stop 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/2965700635 shutdown_connections 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- 172.21.15.23:0/2965700635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 0x7f208c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- 172.21.15.23:0/2965700635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c101910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- 172.21.15.23:0/2965700635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 0x7f208c10a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/2965700635 >> 172.21.15.23:0/2965700635 conn(0x7f208c0770a0 msgr2=0x7f208c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/2965700635 shutdown_connections 2023-11-17T00:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/2965700635 wait complete. 2023-11-17T00:00:13.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 Processor -- start 2023-11-17T00:00:13.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- start start 2023-11-17T00:00:13.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:13.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 0x7f208c1947c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:13.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 0x7f208c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c198240 con 0x7f208c10a890 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c1983b0 con 0x7f208c101e50 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c198570 con 0x7f208c101540 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:35464/0 (socket says 172.21.15.23:35464) 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 -- 172.21.15.23:0/3288937989 learned_addr learned my addr 172.21.15.23:0/3288937989 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:13.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092802700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 0x7f208c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2091800700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 0x7f208c1947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 -- 172.21.15.23:0/3288937989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 msgr2=0x7f208c1947c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 0x7f208c1947c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 -- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 msgr2=0x7f208c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 0x7f208c197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 -- 172.21.15.23:0/3288937989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088009520 con 0x7f208c101540 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2092001700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2088009810 tx=0x7f208800fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208801c070 con 0x7f208c101540 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f208800bba0 con 0x7f208c101540 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c10e210 con 0x7f208c101540 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.013+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880049c0 con 0x7f208c101540 2023-11-17T00:00:13.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.019+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c10e080 con 0x7f208c101540 2023-11-17T00:00:13.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.020+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2088020070 con 0x7f208c101540 2023-11-17T00:00:13.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.020+0000 7f2082ffd700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 0x7f20780a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:13.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.020+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f20880cd9e0 con 0x7f208c101540 2023-11-17T00:00:13.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.020+0000 7f2091800700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 0x7f20780a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:13.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.020+0000 7f2091800700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 0x7f20780a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f208c195860 tx=0x7f207c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:13.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.027+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c10e3a0 con 0x7f208c101540 2023-11-17T00:00:13.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.034+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2088096e10 con 0x7f208c101540 2023-11-17T00:00:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.252+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f208c066ea0 con 0x7f208c101540 2023-11-17T00:00:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.253+0000 7f2082ffd700 1 -- 172.21.15.23:0/3288937989 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f2088027020 con 0x7f208c101540 2023-11-17T00:00:13.253 INFO:teuthology.orchestra.run.smithi023.stdout:304942678049 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 msgr2=0x7f20780a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 0x7f20780a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f208c195860 tx=0x7f207c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 msgr2=0x7f208c194240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2088009810 tx=0x7f208800fa70 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 shutdown_connections 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f208c10a890 0x7f208c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f20780a11c0 0x7f20780a3670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f208c101540 0x7f208c194240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 --2- 172.21.15.23:0/3288937989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f208c101e50 0x7f208c1947c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 >> 172.21.15.23:0/3288937989 conn(0x7f208c0770a0 msgr2=0x7f208c0fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:13.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 shutdown_connections 2023-11-17T00:00:13.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:13.255+0000 7f2094265700 1 -- 172.21.15.23:0/3288937989 wait complete. 2023-11-17T00:00:13.511 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920832 for osd.4 2023-11-17T00:00:13.511 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:14.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:14 smithi023 ceph-mon[97824]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:14.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:14 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3288937989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-17T00:00:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:14 smithi173 ceph-mon[131534]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:14 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3288937989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-17T00:00:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:14 smithi156 ceph-mon[131105]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:00:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:14 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3288937989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-17T00:00:14.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678047 got 304942678049 for osd.10 2023-11-17T00:00:14.723 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:14.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.723+0000 7f6e7678a700 1 -- 172.21.15.23:0/2648128128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7010a890 msgr2=0x7f6e7010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:14.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.723+0000 7f6e7678a700 1 --2- 172.21.15.23:0/2648128128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7010a890 0x7f6e7010ccd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800b0a0 tx=0x7f6e6800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 -- 172.21.15.23:0/2648128128 shutdown_connections 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 --2- 172.21.15.23:0/2648128128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7010a890 0x7f6e7010ccd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 --2- 172.21.15.23:0/2648128128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 0x7f6e70101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 --2- 172.21.15.23:0/2648128128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e70101e50 0x7f6e7010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 -- 172.21.15.23:0/2648128128 >> 172.21.15.23:0/2648128128 conn(0x7f6e700770a0 msgr2=0x7f6e700774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 -- 172.21.15.23:0/2648128128 shutdown_connections 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.724+0000 7f6e7678a700 1 -- 172.21.15.23:0/2648128128 wait complete. 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.725+0000 7f6e7678a700 1 Processor -- start 2023-11-17T00:00:14.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.725+0000 7f6e7678a700 1 -- start start 2023-11-17T00:00:14.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.725+0000 7f6e7678a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 0x7f6e70198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:14.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.725+0000 7f6e7678a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:14.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e7678a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 0x7f6e7019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e7678a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7019c600 con 0x7f6e70101e50 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e7678a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7019c770 con 0x7f6e7010a890 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e7678a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7019c930 con 0x7f6e70101540 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53508/0 (socket says 172.21.15.23:53508) 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 -- 172.21.15.23:0/1702301021 learned_addr learned my addr 172.21.15.23:0/1702301021 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6ffff700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 0x7f6e70198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e74d27700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 0x7f6e7019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 -- 172.21.15.23:0/1702301021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 msgr2=0x7f6e70198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 0x7f6e70198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 -- 172.21.15.23:0/1702301021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 msgr2=0x7f6e7019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 0x7f6e7019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e6f7fe700 1 -- 172.21.15.23:0/1702301021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68009d00 con 0x7f6e70101e50 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.726+0000 7f6e74d27700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 0x7f6e7019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e6f7fe700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400e9a0 tx=0x7f6e6400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64009790 con 0x7f6e70101e50 2023-11-17T00:00:14.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70110350 con 0x7f6e70101e50 2023-11-17T00:00:14.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e64010430 con 0x7f6e70101e50 2023-11-17T00:00:14.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e701101c0 con 0x7f6e70101e50 2023-11-17T00:00:14.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.727+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64009790 con 0x7f6e70101e50 2023-11-17T00:00:14.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.729+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e701104e0 con 0x7f6e70101e50 2023-11-17T00:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.731+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e640108a0 con 0x7f6e70101e50 2023-11-17T00:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.732+0000 7f6e6d7fa700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 0x7f6e580a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:14.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.732+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e64014070 con 0x7f6e70101e50 2023-11-17T00:00:14.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.733+0000 7f6e6ffff700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 0x7f6e580a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:14.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.733+0000 7f6e6ffff700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 0x7f6e580a36f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009c60 tx=0x7f6e60009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:14.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:14.737+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e64051730 con 0x7f6e70101e50 2023-11-17T00:00:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:14.915 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.033+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6e70066ea0 con 0x7f6e70101e50 2023-11-17T00:00:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.034+0000 7f6e6d7fa700 1 -- 172.21.15.23:0/1702301021 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6e64095df0 con 0x7f6e70101e50 2023-11-17T00:00:15.035 INFO:teuthology.orchestra.run.smithi023.stdout:197568495671 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 msgr2=0x7f6e580a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 0x7f6e580a36f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009c60 tx=0x7f6e60009260 comp rx=0 tx=0).stop 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 msgr2=0x7f6e70198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400e9a0 tx=0x7f6e6400ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 shutdown_connections 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e7010a890 0x7f6e7019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6e580a1240 0x7f6e580a36f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70101540 0x7f6e70198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 --2- 172.21.15.23:0/1702301021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e70101e50 0x7f6e70198c40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.036+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 >> 172.21.15.23:0/1702301021 conn(0x7f6e700770a0 msgr2=0x7f6e700fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.037+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 shutdown_connections 2023-11-17T00:00:15.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.037+0000 7f6e7678a700 1 -- 172.21.15.23:0/1702301021 wait complete. 2023-11-17T00:00:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.197+0000 7f1d21667700 1 -- 172.21.15.23:0/640688845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c1055e0 msgr2=0x7f1d1c10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.197+0000 7f1d21667700 1 --2- 172.21.15.23:0/640688845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c10d160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400b0d0 tx=0x7f1d1400b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 -- 172.21.15.23:0/640688845 shutdown_connections 2023-11-17T00:00:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 --2- 172.21.15.23:0/640688845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c10d160 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 --2- 172.21.15.23:0/640688845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100540 0x7f1d1c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 --2- 172.21.15.23:0/640688845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 -- 172.21.15.23:0/640688845 >> 172.21.15.23:0/640688845 conn(0x7f1d1c0fbdc0 msgr2=0x7f1d1c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 -- 172.21.15.23:0/640688845 shutdown_connections 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.198+0000 7f1d21667700 1 -- 172.21.15.23:0/640688845 wait complete. 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.199+0000 7f1d21667700 1 Processor -- start 2023-11-17T00:00:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.199+0000 7f1d21667700 1 -- start start 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c100540 0x7f1d1c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c198c10 con 0x7f1d1c100540 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c198d80 con 0x7f1d1c1055e0 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d21667700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c198f40 con 0x7f1d1c100ee0 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:35506/0 (socket says 172.21.15.23:35506) 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 -- 172.21.15.23:0/3381243487 learned_addr learned my addr 172.21.15.23:0/3381243487 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1b7fe700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 -- 172.21.15.23:0/3381243487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c1055e0 msgr2=0x7f1d1c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 -- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c100540 msgr2=0x7f1d1c19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1affd700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c100540 0x7f1d1c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c100540 0x7f1d1c19e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.200+0000 7f1d1a7fc700 1 -- 172.21.15.23:0/3381243487 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d14009d00 con 0x7f1d1c100ee0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d1a7fc700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e970 tx=0x7f1d1000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10009790 con 0x7f1d1c100ee0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d1000eec0 con 0x7f1d1c100ee0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10005590 con 0x7f1d1c100ee0 2023-11-17T00:00:15.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c110370 con 0x7f1d1c100ee0 2023-11-17T00:00:15.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.201+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c1101e0 con 0x7f1d1c100ee0 2023-11-17T00:00:15.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.202+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c110500 con 0x7f1d1c100ee0 2023-11-17T00:00:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.204+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1d1000cbd0 con 0x7f1d1c100ee0 2023-11-17T00:00:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.206+0000 7f1d03fff700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 0x7f1d040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.206+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1d10014070 con 0x7f1d1c100ee0 2023-11-17T00:00:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.206+0000 7f1d1affd700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 0x7f1d040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.207+0000 7f1d1affd700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 0x7f1d040a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009b30 tx=0x7f1d0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.208+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1d10096940 con 0x7f1d1c100ee0 2023-11-17T00:00:15.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:15 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1702301021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T00:00:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.475+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1d1c066ea0 con 0x7f1d1c100ee0 2023-11-17T00:00:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.476+0000 7f1d03fff700 1 -- 172.21.15.23:0/3381243487 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d10004ef0 con 0x7f1d1c100ee0 2023-11-17T00:00:15.477 INFO:teuthology.orchestra.run.smithi023.stdout:77309411405 2023-11-17T00:00:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.478+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 msgr2=0x7f1d040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.478+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 0x7f1d040a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009b30 tx=0x7f1d0c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.478+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 msgr2=0x7f1d1c19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.478+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e970 tx=0x7f1d1000ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 shutdown_connections 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d1c1055e0 0x7f1d1c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1c100540 0x7f1d1c19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f1d040a11e0 0x7f1d040a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 --2- 172.21.15.23:0/3381243487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d1c100ee0 0x7f1d1c19ea60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 >> 172.21.15.23:0/3381243487 conn(0x7f1d1c0fbdc0 msgr2=0x7f1d1c10b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 shutdown_connections 2023-11-17T00:00:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.479+0000 7f1d21667700 1 -- 172.21.15.23:0/3381243487 wait complete. 2023-11-17T00:00:15.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:15 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1702301021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T00:00:15.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:15 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1702301021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T00:00:15.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495671 for osd.6 2023-11-17T00:00:15.736 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 -- 172.21.15.23:0/635746659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0104310 msgr2=0x7f99c010be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/635746659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0104310 0x7f99c010be90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f99b800b0d0 tx=0x7f99b800b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 -- 172.21.15.23:0/635746659 shutdown_connections 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/635746659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0104310 0x7f99c010be90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/635746659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c00ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/635746659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c00ffc10 0x7f99c0103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 -- 172.21.15.23:0/635746659 >> 172.21.15.23:0/635746659 conn(0x7f99c00761a0 msgr2=0x7f99c00765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 -- 172.21.15.23:0/635746659 shutdown_connections 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.310+0000 7f99c6e1a700 1 -- 172.21.15.23:0/635746659 wait complete. 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 Processor -- start 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 -- start start 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c01963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c0104310 0x7f99c019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019b390 con 0x7f99c00ffc10 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019b500 con 0x7f99c0104310 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.312+0000 7f99c6e1a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019b6c0 con 0x7f99c00ff270 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5617700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5617700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53538/0 (socket says 172.21.15.23:53538) 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5617700 1 -- 172.21.15.23:0/708850662 learned_addr learned my addr 172.21.15.23:0/708850662 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c01963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c6619700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c0104310 0x7f99c019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 -- 172.21.15.23:0/708850662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c0104310 msgr2=0x7f99c019ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c0104310 0x7f99c019ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 -- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 msgr2=0x7f99c0196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5e18700 1 -- 172.21.15.23:0/708850662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8009d00 con 0x7f99c00ff270 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.313+0000 7f99c5617700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99c5e18700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c01963d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009c50 tx=0x7f99bc009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc015070 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99bc019410 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c010f0a0 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc018610 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.314+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c010ef10 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.315+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c010f230 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.317+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f99bc019880 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.318+0000 7f99b6ffd700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 0x7f99ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.319+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f99bc0d0390 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.319+0000 7f99c5617700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 0x7f99ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.319+0000 7f99c5617700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 0x7f99ac0a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f99c01979f0 tx=0x7f99b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.322+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f99bc099790 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.560+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f99c004e8c0 con 0x7f99c00ff270 2023-11-17T00:00:15.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.561+0000 7f99b6ffd700 1 -- 172.21.15.23:0/708850662 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f99bc09cdb0 con 0x7f99c00ff270 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stdout:163208757310 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 msgr2=0x7f99ac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 0x7f99ac0a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f99c01979f0 tx=0x7f99b0009260 comp rx=0 tx=0).stop 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 msgr2=0x7f99c01963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c01963d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009c50 tx=0x7f99bc009f60 comp rx=0 tx=0).stop 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 shutdown_connections 2023-11-17T00:00:15.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99c0104310 0x7f99c019ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f99ac0a11e0 0x7f99ac0a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99c00ff270 0x7f99c01963d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 --2- 172.21.15.23:0/708850662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c00ffc10 0x7f99c0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.563+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 >> 172.21.15.23:0/708850662 conn(0x7f99c00761a0 msgr2=0x7f99c00685b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.564+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 shutdown_connections 2023-11-17T00:00:15.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.564+0000 7f99c6e1a700 1 -- 172.21.15.23:0/708850662 wait complete. 2023-11-17T00:00:15.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 -- 172.21.15.23:0/735331913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 msgr2=0x7f2d4410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/735331913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4410ccd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009f90 tx=0x7f2d3c00b330 comp rx=0 tx=0).stop 2023-11-17T00:00:15.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 -- 172.21.15.23:0/735331913 shutdown_connections 2023-11-17T00:00:15.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/735331913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4410ccd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/735331913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101540 0x7f2d44101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/735331913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101e50 0x7f2d4410a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 -- 172.21.15.23:0/735331913 >> 172.21.15.23:0/735331913 conn(0x7f2d440770a0 msgr2=0x7f2d440774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 -- 172.21.15.23:0/735331913 shutdown_connections 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.764+0000 7f2d4acb6700 1 -- 172.21.15.23:0/735331913 wait complete. 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.765+0000 7f2d4acb6700 1 Processor -- start 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 -- start start 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101e50 0x7f2d44198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4419c5e0 con 0x7f2d4410a890 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4419c750 con 0x7f2d44101540 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d4acb6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4419c910 con 0x7f2d44101e50 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d49253700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d48a52700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d48a52700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:51678/0 (socket says 172.21.15.23:51678) 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d48a52700 1 -- 172.21.15.23:0/996820330 learned_addr learned my addr 172.21.15.23:0/996820330 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.766+0000 7f2d49253700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53562/0 (socket says 172.21.15.23:53562) 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d43fff700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101e50 0x7f2d44198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 -- 172.21.15.23:0/996820330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101e50 msgr2=0x7f2d44198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101e50 0x7f2d44198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 -- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 msgr2=0x7f2d4419bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 -- 172.21.15.23:0/996820330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c009c40 con 0x7f2d44101540 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d49253700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d48a52700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800e9c0 tx=0x7f2d3800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800cbd0 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d3800eeb0 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d44110350 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.767+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38005580 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.768+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d441101c0 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.769+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d441104e0 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.771+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2d38004750 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.772+0000 7f2d41ffb700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 0x7f2d340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.772+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2d38014070 con 0x7f2d44101540 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.772+0000 7f2d43fff700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 0x7f2d340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.773+0000 7f2d43fff700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 0x7f2d340a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2d44199d00 tx=0x7f2d30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.774+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2d38096820 con 0x7f2d44101540 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 -- 172.21.15.23:0/1743220541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 msgr2=0x7f56c010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 --2- 172.21.15.23:0/1743220541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c010ee70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b0a0 tx=0x7f56b800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 -- 172.21.15.23:0/1743220541 shutdown_connections 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 --2- 172.21.15.23:0/1743220541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c010ee70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 --2- 172.21.15.23:0/1743220541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 0x7f56c00ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 --2- 172.21.15.23:0/1743220541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 0x7f56c010c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 -- 172.21.15.23:0/1743220541 >> 172.21.15.23:0/1743220541 conn(0x7f56c00761a0 msgr2=0x7f56c00765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 -- 172.21.15.23:0/1743220541 shutdown_connections 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.815+0000 7f56c5830700 1 -- 172.21.15.23:0/1743220541 wait complete. 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.817+0000 7f56c5830700 1 Processor -- start 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.817+0000 7f56c5830700 1 -- start start 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 0x7f56c0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 0x7f56c0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c019d510 con 0x7f56c010ca90 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c019d680 con 0x7f56c00feee0 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c5830700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c019d840 con 0x7f56c00ff7f0 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56bffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 0x7f56c0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53582/0 (socket says 172.21.15.23:53582) 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 -- 172.21.15.23:0/12183099 learned_addr learned my addr 172.21.15.23:0/12183099 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c482e700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 0x7f56c0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 -- 172.21.15.23:0/12183099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 msgr2=0x7f56c0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 0x7f56c0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 -- 172.21.15.23:0/12183099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 msgr2=0x7f56c0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.818+0000 7f56c502f700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 0x7f56c0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56c502f700 1 -- 172.21.15.23:0/12183099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b8009d00 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56c502f700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fd0 tx=0x7f56b8007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800e040 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b801d470 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b80052a0 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c0112380 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.819+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c01121f0 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.820+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c0112510 con 0x7f56c010ca90 2023-11-17T00:00:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.823+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f56b801d5e0 con 0x7f56c010ca90 2023-11-17T00:00:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.825+0000 7f56bdffb700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 0x7f56b00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.825+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f56b80ce740 con 0x7f56c010ca90 2023-11-17T00:00:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.825+0000 7f56c482e700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 0x7f56b00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.826+0000 7f56c482e700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 0x7f56b00a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007be0 tx=0x7f56b40073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.826+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f56b8097b70 con 0x7f56c010ca90 2023-11-17T00:00:15.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.995+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2d44066ea0 con 0x7f2d44101540 2023-11-17T00:00:15.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.996+0000 7f2d41ffb700 1 -- 172.21.15.23:0/996820330 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2d38099e40 con 0x7f2d44101540 2023-11-17T00:00:15.996 INFO:teuthology.orchestra.run.smithi023.stdout:219043332146 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 msgr2=0x7f2d340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 0x7f2d340a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2d44199d00 tx=0x7f2d30009260 comp rx=0 tx=0).stop 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 msgr2=0x7f2d441986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800e9c0 tx=0x7f2d3800ecd0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 shutdown_connections 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d4410a890 0x7f2d4419bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d44101540 0x7f2d441986e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2d340a11e0 0x7f2d340a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 --2- 172.21.15.23:0/996820330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d44101e50 0x7f2d44198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 >> 172.21.15.23:0/996820330 conn(0x7f2d440770a0 msgr2=0x7f2d440fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:16.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 shutdown_connections 2023-11-17T00:00:16.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:15.998+0000 7f2d4acb6700 1 -- 172.21.15.23:0/996820330 wait complete. 2023-11-17T00:00:16.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.053+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f56c004e8c0 con 0x7f56c010ca90 2023-11-17T00:00:16.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.053+0000 7f56bdffb700 1 -- 172.21.15.23:0/12183099 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f56b80253b0 con 0x7f56c010ca90 2023-11-17T00:00:16.058 INFO:teuthology.orchestra.run.smithi023.stdout:249108103215 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.055+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 msgr2=0x7f56b00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.055+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 0x7f56b00a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007be0 tx=0x7f56b40073d0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.055+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 msgr2=0x7f56c019cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.055+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fd0 tx=0x7f56b8007930 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 shutdown_connections 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c010ca90 0x7f56c019cf00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56c00feee0 0x7f56c0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f56b00a1260 0x7f56b00a3710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 --2- 172.21.15.23:0/12183099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c00ff7f0 0x7f56c0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 >> 172.21.15.23:0/12183099 conn(0x7f56c00761a0 msgr2=0x7f56c00fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:16.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 shutdown_connections 2023-11-17T00:00:16.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:16.056+0000 7f56c5830700 1 -- 172.21.15.23:0/12183099 wait complete. 2023-11-17T00:00:16.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:16 smithi023 ceph-mon[97824]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-11-17T00:00:16.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3381243487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T00:00:16.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/708850662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T00:00:16.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/996820330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T00:00:16.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:16 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/12183099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:16 smithi173 ceph-mon[131534]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3381243487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/708850662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/996820330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:16 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/12183099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-17T00:00:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:16 smithi156 ceph-mon[131105]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-11-17T00:00:16.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3381243487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T00:00:16.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/708850662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T00:00:16.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/996820330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T00:00:16.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:16 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/12183099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-17T00:00:16.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411405 for osd.2 2023-11-17T00:00:16.938 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:17.048 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757308 got 163208757310 for osd.5 2023-11-17T00:00:17.048 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:17.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103212 got 249108103215 for osd.8 2023-11-17T00:00:17.084 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:17.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 -- 172.21.15.23:0/2622665592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100540 msgr2=0x7f614c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:17.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2622665592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100540 0x7f614c100910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f61400099c0 tx=0x7f6140009cd0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 -- 172.21.15.23:0/2622665592 shutdown_connections 2023-11-17T00:00:17.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2622665592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c1055e0 0x7f614c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2622665592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100540 0x7f614c100910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2622665592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c100ee0 0x7f614c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 -- 172.21.15.23:0/2622665592 >> 172.21.15.23:0/2622665592 conn(0x7f614c0fbdc0 msgr2=0x7f614c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:17.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 -- 172.21.15.23:0/2622665592 shutdown_connections 2023-11-17T00:00:17.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.320+0000 7f6152c5a700 1 -- 172.21.15.23:0/2622665592 wait complete. 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.321+0000 7f6152c5a700 1 Processor -- start 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.321+0000 7f6152c5a700 1 -- start start 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 0x7f614c198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c1055e0 0x7f614c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c19d5c0 con 0x7f614c100ee0 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c19d730 con 0x7f614c100540 2023-11-17T00:00:17.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f6152c5a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c19d8f0 con 0x7f614c1055e0 2023-11-17T00:00:17.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f61509f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:17.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f614bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 0x7f614c198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:17.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f61509f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:51708/0 (socket says 172.21.15.23:51708) 2023-11-17T00:00:17.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f61509f6700 1 -- 172.21.15.23:0/2674372479 learned_addr learned my addr 172.21.15.23:0/2674372479 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:17.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.322+0000 7f61511f7700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c1055e0 0x7f614c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:17.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 -- 172.21.15.23:0/2674372479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c1055e0 msgr2=0x7f614c19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:17.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c1055e0 0x7f614c19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 -- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 msgr2=0x7f614c198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 0x7f614c198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 -- 172.21.15.23:0/2674372479 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61400095f0 con 0x7f614c100540 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f614bfff700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 0x7f614c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f61509f6700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6140009990 tx=0x7f6140005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614001d070 con 0x7f614c100540 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6140003f90 con 0x7f614c100540 2023-11-17T00:00:17.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c110370 con 0x7f614c100540 2023-11-17T00:00:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140022ba0 con 0x7f614c100540 2023-11-17T00:00:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.323+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c1101e0 con 0x7f614c100540 2023-11-17T00:00:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.324+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c110500 con 0x7f614c100540 2023-11-17T00:00:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.326+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6140003ad0 con 0x7f614c100540 2023-11-17T00:00:17.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.328+0000 7f6149ffb700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 0x7f613c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:17.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.328+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f61400ce140 con 0x7f614c100540 2023-11-17T00:00:17.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.328+0000 7f614bfff700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 0x7f613c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:17.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.328+0000 7f614bfff700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 0x7f613c0a3710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f614c199c20 tx=0x7f613800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:17.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.332+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6140097570 con 0x7f614c100540 2023-11-17T00:00:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.543+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f614c066ea0 con 0x7f614c100540 2023-11-17T00:00:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.543+0000 7f6149ffb700 1 -- 172.21.15.23:0/2674372479 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f614009ab90 con 0x7f614c100540 2023-11-17T00:00:17.544 INFO:teuthology.orchestra.run.smithi023.stdout:339302416414 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 msgr2=0x7f613c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 0x7f613c0a3710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f614c199c20 tx=0x7f613800b410 comp rx=0 tx=0).stop 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 msgr2=0x7f614c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6140009990 tx=0x7f6140005740 comp rx=0 tx=0).stop 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 shutdown_connections 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f613c0a1260 0x7f613c0a3710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f614c1055e0 0x7f614c19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f614c100540 0x7f614c198600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 --2- 172.21.15.23:0/2674372479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c100ee0 0x7f614c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:17.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 >> 172.21.15.23:0/2674372479 conn(0x7f614c0fbdc0 msgr2=0x7f614c0fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:17.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 shutdown_connections 2023-11-17T00:00:17.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:17.546+0000 7f6152c5a700 1 -- 172.21.15.23:0/2674372479 wait complete. 2023-11-17T00:00:17.659 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332144 got 219043332146 for osd.7 2023-11-17T00:00:17.659 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:18.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:18 smithi023 ceph-mon[97824]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:18.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:18 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2674372479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-17T00:00:18.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 339302416411 got 339302416414 for osd.11 2023-11-17T00:00:18.491 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:18 smithi173 ceph-mon[131534]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:18 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2674372479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-17T00:00:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:18 smithi156 ceph-mon[131105]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:18 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2674372479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-17T00:00:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.824+0000 7fe26c623700 1 -- 172.21.15.23:0/3317974190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2641055e0 msgr2=0x7fe26410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.824+0000 7fe26c623700 1 --2- 172.21.15.23:0/3317974190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2641055e0 0x7fe26410d160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b0d0 tx=0x7fe25c00b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 -- 172.21.15.23:0/3317974190 shutdown_connections 2023-11-17T00:00:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 --2- 172.21.15.23:0/3317974190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2641055e0 0x7fe26410d160 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 --2- 172.21.15.23:0/3317974190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 0x7fe264100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 --2- 172.21.15.23:0/3317974190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe264100ee0 0x7fe264104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 -- 172.21.15.23:0/3317974190 >> 172.21.15.23:0/3317974190 conn(0x7fe2640fbdc0 msgr2=0x7fe2640fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 -- 172.21.15.23:0/3317974190 shutdown_connections 2023-11-17T00:00:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.825+0000 7fe26c623700 1 -- 172.21.15.23:0/3317974190 wait complete. 2023-11-17T00:00:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.826+0000 7fe26c623700 1 Processor -- start 2023-11-17T00:00:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.826+0000 7fe26c623700 1 -- start start 2023-11-17T00:00:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 0x7fe264198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2641055e0 0x7fe26419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26419c5c0 con 0x7fe264100ee0 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26419c730 con 0x7fe264100540 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26c623700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26419c8f0 con 0x7fe2641055e0 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53616/0 (socket says 172.21.15.23:53616) 2023-11-17T00:00:18.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 -- 172.21.15.23:0/3394134717 learned_addr learned my addr 172.21.15.23:0/3394134717 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:18.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26a3bf700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 0x7fe264198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:18.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe26abc0700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2641055e0 0x7fe26419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:18.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 -- 172.21.15.23:0/3394134717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2641055e0 msgr2=0x7fe26419bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:18.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2641055e0 0x7fe26419bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 -- 172.21.15.23:0/3394134717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 msgr2=0x7fe264198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:18.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 0x7fe264198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:18.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.827+0000 7fe269bbe700 1 -- 172.21.15.23:0/3394134717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c009d00 con 0x7fe264100ee0 2023-11-17T00:00:18.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe269bbe700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e970 tx=0x7fe25400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:18.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254009790 con 0x7fe264100ee0 2023-11-17T00:00:18.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe25400c7e0 con 0x7fe264100ee0 2023-11-17T00:00:18.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe264110370 con 0x7fe264100ee0 2023-11-17T00:00:18.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2540054e0 con 0x7fe264100ee0 2023-11-17T00:00:18.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.828+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2641101e0 con 0x7fe264100ee0 2023-11-17T00:00:18.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.829+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe264110500 con 0x7fe264100ee0 2023-11-17T00:00:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.832+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe25400c950 con 0x7fe264100ee0 2023-11-17T00:00:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.833+0000 7fe25b7fe700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 0x7fe2500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.833+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe254014070 con 0x7fe264100ee0 2023-11-17T00:00:18.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.833+0000 7fe26a3bf700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 0x7fe2500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:18.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.834+0000 7fe26a3bf700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 0x7fe2500a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe260005950 tx=0x7fe26000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:18.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:18.837+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe254096260 con 0x7fe264100ee0 2023-11-17T00:00:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.052+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe2641a57d0 con 0x7fe264100ee0 2023-11-17T00:00:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.052+0000 7fe25b7fe700 1 -- 172.21.15.23:0/3394134717 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe254099880 con 0x7fe264100ee0 2023-11-17T00:00:19.053 INFO:teuthology.orchestra.run.smithi023.stdout:55834574931 2023-11-17T00:00:19.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 msgr2=0x7fe2500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 0x7fe2500a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe260005950 tx=0x7fe26000b410 comp rx=0 tx=0).stop 2023-11-17T00:00:19.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 msgr2=0x7fe264198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe25400e970 tx=0x7fe25400ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:19.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 shutdown_connections 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2500a11e0 0x7fe2500a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.054+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2641055e0 0x7fe26419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.055+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe264100540 0x7fe264198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.055+0000 7fe26c623700 1 --2- 172.21.15.23:0/3394134717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe264100ee0 0x7fe264198c50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.055+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 >> 172.21.15.23:0/3394134717 conn(0x7fe2640fbdc0 msgr2=0x7fe26410b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.055+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 shutdown_connections 2023-11-17T00:00:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.055+0000 7fe26c623700 1 -- 172.21.15.23:0/3394134717 wait complete. 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.233+0000 7ff66cd60700 1 -- 172.21.15.23:0/593082398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6681055e0 msgr2=0x7ff66810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.233+0000 7ff66cd60700 1 --2- 172.21.15.23:0/593082398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6681055e0 0x7ff66810d160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff660009fc0 tx=0x7ff66000b330 comp rx=0 tx=0).stop 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 -- 172.21.15.23:0/593082398 shutdown_connections 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 --2- 172.21.15.23:0/593082398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6681055e0 0x7ff66810d160 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 --2- 172.21.15.23:0/593082398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 --2- 172.21.15.23:0/593082398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff668100ee0 0x7ff668104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 -- 172.21.15.23:0/593082398 >> 172.21.15.23:0/593082398 conn(0x7ff6680fbdc0 msgr2=0x7ff6680fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 -- 172.21.15.23:0/593082398 shutdown_connections 2023-11-17T00:00:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.234+0000 7ff66cd60700 1 -- 172.21.15.23:0/593082398 wait complete. 2023-11-17T00:00:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 Processor -- start 2023-11-17T00:00:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 -- start start 2023-11-17T00:00:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6681055e0 0x7ff6681a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff668073c20 con 0x7ff668100ee0 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681a4c60 con 0x7ff668100540 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.235+0000 7ff66cd60700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681a4dd0 con 0x7ff6681055e0 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff665d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:19.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:51738/0 (socket says 172.21.15.23:51738) 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff665d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53640/0 (socket says 172.21.15.23:53640) 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 -- 172.21.15.23:0/4118406101 learned_addr learned my addr 172.21.15.23:0/4118406101 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff666d9d700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6681055e0 0x7ff6681a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 -- 172.21.15.23:0/4118406101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6681055e0 msgr2=0x7ff6681a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6681055e0 0x7ff6681a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 -- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 msgr2=0x7ff668073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff66659c700 1 -- 172.21.15.23:0/4118406101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff660009c40 con 0x7ff668100540 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.236+0000 7ff665d9b700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff66659c700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65800e9a0 tx=0x7ff65800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff658009790 con 0x7ff668100540 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff65800ce80 con 0x7ff668100540 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff668110370 con 0x7ff668100540 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff658018680 con 0x7ff668100540 2023-11-17T00:00:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.237+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6681101e0 con 0x7ff668100540 2023-11-17T00:00:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.238+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff668110500 con 0x7ff668100540 2023-11-17T00:00:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.240+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff658010430 con 0x7ff668100540 2023-11-17T00:00:19.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.241+0000 7ff6577fe700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 0x7ff6500abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:19.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.242+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff658014070 con 0x7ff668100540 2023-11-17T00:00:19.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.242+0000 7ff665d9b700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 0x7ff6500abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:19.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.242+0000 7ff665d9b700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 0x7ff6500abea0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff6680745e0 tx=0x7ff65c009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:19.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.244+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff6580963a0 con 0x7ff668100540 2023-11-17T00:00:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.451+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff668066ea0 con 0x7ff668100540 2023-11-17T00:00:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.451+0000 7ff6577fe700 1 -- 172.21.15.23:0/4118406101 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff6580999c0 con 0x7ff668100540 2023-11-17T00:00:19.452 INFO:teuthology.orchestra.run.smithi023.stdout:107374182472 2023-11-17T00:00:19.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.453+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 msgr2=0x7ff6500abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.453+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 0x7ff6500abea0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff6680745e0 tx=0x7ff65c009310 comp rx=0 tx=0).stop 2023-11-17T00:00:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.453+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 msgr2=0x7ff668072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65800e9a0 tx=0x7ff65800ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 shutdown_connections 2023-11-17T00:00:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff6500a99f0 0x7ff6500abea0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6681055e0 0x7ff6681a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff668100540 0x7ff668072df0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 --2- 172.21.15.23:0/4118406101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff668100ee0 0x7ff668073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 >> 172.21.15.23:0/4118406101 conn(0x7ff6680fbdc0 msgr2=0x7ff6680fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 shutdown_connections 2023-11-17T00:00:19.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:19.454+0000 7ff66cd60700 1 -- 172.21.15.23:0/4118406101 wait complete. 2023-11-17T00:00:19.504 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574928 got 55834574931 for osd.1 2023-11-17T00:00:19.504 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:19.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3394134717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T00:00:19.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:19 smithi023 ceph-mon[97824]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:19.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:19 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4118406101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T00:00:19.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182469 got 107374182472 for osd.3 2023-11-17T00:00:19.995 DEBUG:teuthology.parallel:result is None 2023-11-17T00:00:19.995 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-17T00:00:19.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T00:00:20.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3394134717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T00:00:20.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:19 smithi173 ceph-mon[131534]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:20.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:19 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4118406101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T00:00:20.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3394134717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T00:00:20.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:19 smithi156 ceph-mon[131105]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-11-17T00:00:20.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:19 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4118406101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T00:00:20.880 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 -- 172.21.15.23:0/567062023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63501055e0 msgr2=0x7f635010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 --2- 172.21.15.23:0/567062023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63501055e0 0x7f635010d160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f634800b0d0 tx=0x7f634800b3e0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 -- 172.21.15.23:0/567062023 shutdown_connections 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 --2- 172.21.15.23:0/567062023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63501055e0 0x7f635010d160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 --2- 172.21.15.23:0/567062023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350100540 0x7f6350100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 --2- 172.21.15.23:0/567062023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100ee0 0x7f6350104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 -- 172.21.15.23:0/567062023 >> 172.21.15.23:0/567062023 conn(0x7f63500fbdc0 msgr2=0x7f63500fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:22.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 -- 172.21.15.23:0/567062023 shutdown_connections 2023-11-17T00:00:22.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.405+0000 7f635613d700 1 -- 172.21.15.23:0/567062023 wait complete. 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.406+0000 7f635613d700 1 Processor -- start 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.406+0000 7f635613d700 1 -- start start 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.406+0000 7f635613d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f635613d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6350100ee0 0x7f6350073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f635613d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 0x7f63501a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f635613d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350073c20 con 0x7f6350100ee0 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f635613d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63501a4c60 con 0x7f63501055e0 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f635613d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63501a4dd0 con 0x7f6350100540 2023-11-17T00:00:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6350100ee0 0x7f6350073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:56320/0 (socket says 172.21.15.23:56320) 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 -- 172.21.15.23:0/128527777 learned_addr learned my addr 172.21.15.23:0/128527777 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634ffff700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 0x7f63501a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 -- 172.21.15.23:0/128527777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 msgr2=0x7f63501a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 0x7f63501a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 -- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6350100ee0 msgr2=0x7f6350073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6350100ee0 0x7f6350073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634f7fe700 1 -- 172.21.15.23:0/128527777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348009d00 con 0x7f6350100540 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.407+0000 7f634ffff700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 0x7f63501a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f634f7fe700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9a0 tx=0x7f634000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340009790 con 0x7f6350100540 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f634000cbd0 con 0x7f6350100540 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6350110370 con 0x7f6350100540 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340018680 con 0x7f6350100540 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.408+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63501101e0 con 0x7f6350100540 2023-11-17T00:00:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.409+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350110500 con 0x7f6350100540 2023-11-17T00:00:22.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.411+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f634000cd40 con 0x7f6350100540 2023-11-17T00:00:22.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.413+0000 7f634cff9700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 0x7f63380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:22.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.413+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6340014070 con 0x7f6350100540 2023-11-17T00:00:22.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.413+0000 7f634effd700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 0x7f63380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:22.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.413+0000 7f634effd700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 0x7f63380a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63500745e0 tx=0x7f6344009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:22.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.414+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6340096570 con 0x7f6350100540 2023-11-17T00:00:22.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:22 smithi023 ceph-mon[97824]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:22 smithi173 ceph-mon[131534]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:22 smithi156 ceph-mon[131105]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:22.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.616+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63500fce30 con 0x7f63380a11e0 2023-11-17T00:00:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.630+0000 7f634cff9700 1 -- 172.21.15.23:0/128527777 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427147 (secure 0 0 0) 0x7f63500fce30 con 0x7f63380a11e0 2023-11-17T00:00:22.638 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.635+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 msgr2=0x7f63380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.635+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 0x7f63380a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63500745e0 tx=0x7f6344009260 comp rx=0 tx=0).stop 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 msgr2=0x7f6350072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9a0 tx=0x7f634000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 shutdown_connections 2023-11-17T00:00:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63501055e0 0x7f63501a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f63380a11e0 0x7f63380a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6350100540 0x7f6350072df0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 --2- 172.21.15.23:0/128527777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6350100ee0 0x7f6350073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 >> 172.21.15.23:0/128527777 conn(0x7f63500fbdc0 msgr2=0x7f63500fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 shutdown_connections 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:22.636+0000 7f635613d700 1 -- 172.21.15.23:0/128527777 wait complete. 2023-11-17T00:00:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-11-17T00:00:23.086 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2023-11-17T00:00:21.078044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654911,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":423,"num_read_kb":327,"num_write":340,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82264,"kb_used_data":13496,"kb_used_omap":0,"kb_used_meta":68672,"kb_avail":1124859560,"statfs":{"total":1151940427776,"available":1151856189440,"internally_reserved":0,"allocated":13819904,"data_stored":7647362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70320128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":44,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241211+0000","last_change":"2023-11-16T23:58:46.280372+0000","last_active":"2023-11-16T23:58:51.241211+0000","last_peered":"2023-11-16T23:58:51.241211+0000","last_clean":"2023-11-16T23:58:51.241211+0000","last_became_active":"2023-11-16T23:58:46.280021+0000","last_became_peered":"2023-11-16T23:58:46.280021+0000","last_unstale":"2023-11-16T23:58:51.241211+0000","last_undegraded":"2023-11-16T23:58:51.241211+0000","last_fullsized":"2023-11-16T23:58:51.241211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:37.900152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243114+0000","last_change":"2023-11-16T23:58:46.297046+0000","last_active":"2023-11-16T23:58:51.243114+0000","last_peered":"2023-11-16T23:58:51.243114+0000","last_clean":"2023-11-16T23:58:51.243114+0000","last_became_active":"2023-11-16T23:58:46.282128+0000","last_became_peered":"2023-11-16T23:58:46.282128+0000","last_unstale":"2023-11-16T23:58:51.243114+0000","last_undegraded":"2023-11-16T23:58:51.243114+0000","last_fullsized":"2023-11-16T23:58:51.243114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:56.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242939+0000","last_change":"2023-11-16T23:58:46.283752+0000","last_active":"2023-11-16T23:58:51.242939+0000","last_peered":"2023-11-16T23:58:51.242939+0000","last_clean":"2023-11-16T23:58:51.242939+0000","last_became_active":"2023-11-16T23:58:46.280787+0000","last_became_peered":"2023-11-16T23:58:46.280787+0000","last_unstale":"2023-11-16T23:58:51.242939+0000","last_undegraded":"2023-11-16T23:58:51.242939+0000","last_fullsized":"2023-11-16T23:58:51.242939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:06.203820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974468+0000","last_change":"2023-11-16T23:58:46.282847+0000","last_active":"2023-11-16T23:58:51.974468+0000","last_peered":"2023-11-16T23:58:51.974468+0000","last_clean":"2023-11-16T23:58:51.974468+0000","last_became_active":"2023-11-16T23:58:46.282527+0000","last_became_peered":"2023-11-16T23:58:46.282527+0000","last_unstale":"2023-11-16T23:58:51.974468+0000","last_undegraded":"2023-11-16T23:58:51.974468+0000","last_fullsized":"2023-11-16T23:58:51.974468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:15.327419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973484+0000","last_change":"2023-11-16T23:58:46.288154+0000","last_active":"2023-11-16T23:58:51.973484+0000","last_peered":"2023-11-16T23:58:51.973484+0000","last_clean":"2023-11-16T23:58:51.973484+0000","last_became_active":"2023-11-16T23:58:46.287752+0000","last_became_peered":"2023-11-16T23:58:46.287752+0000","last_unstale":"2023-11-16T23:58:51.973484+0000","last_undegraded":"2023-11-16T23:58:51.973484+0000","last_fullsized":"2023-11-16T23:58:51.973484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:17.777171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975139+0000","last_change":"2023-11-16T23:58:46.283177+0000","last_active":"2023-11-16T23:58:51.975139+0000","last_peered":"2023-11-16T23:58:51.975139+0000","last_clean":"2023-11-16T23:58:51.975139+0000","last_became_active":"2023-11-16T23:58:46.282753+0000","last_became_peered":"2023-11-16T23:58:46.282753+0000","last_unstale":"2023-11-16T23:58:51.975139+0000","last_undegraded":"2023-11-16T23:58:51.975139+0000","last_fullsized":"2023-11-16T23:58:51.975139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:40.545830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238882+0000","last_change":"2023-11-16T23:58:46.296002+0000","last_active":"2023-11-16T23:58:51.238882+0000","last_peered":"2023-11-16T23:58:51.238882+0000","last_clean":"2023-11-16T23:58:51.238882+0000","last_became_active":"2023-11-16T23:58:46.295191+0000","last_became_peered":"2023-11-16T23:58:46.295191+0000","last_unstale":"2023-11-16T23:58:51.238882+0000","last_undegraded":"2023-11-16T23:58:51.238882+0000","last_fullsized":"2023-11-16T23:58:51.238882+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:32:43.802637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973904+0000","last_change":"2023-11-16T23:58:46.294824+0000","last_active":"2023-11-16T23:58:51.973904+0000","last_peered":"2023-11-16T23:58:51.973904+0000","last_clean":"2023-11-16T23:58:51.973904+0000","last_became_active":"2023-11-16T23:58:46.294502+0000","last_became_peered":"2023-11-16T23:58:46.294502+0000","last_unstale":"2023-11-16T23:58:51.973904+0000","last_undegraded":"2023-11-16T23:58:51.973904+0000","last_fullsized":"2023-11-16T23:58:51.973904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:47:28.853625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974608+0000","last_change":"2023-11-16T23:58:46.292583+0000","last_active":"2023-11-16T23:58:51.974608+0000","last_peered":"2023-11-16T23:58:51.974608+0000","last_clean":"2023-11-16T23:58:51.974608+0000","last_became_active":"2023-11-16T23:58:46.292089+0000","last_became_peered":"2023-11-16T23:58:46.292089+0000","last_unstale":"2023-11-16T23:58:51.974608+0000","last_undegraded":"2023-11-16T23:58:51.974608+0000","last_fullsized":"2023-11-16T23:58:51.974608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:54.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975325+0000","last_change":"2023-11-16T23:58:46.289410+0000","last_active":"2023-11-16T23:58:51.975325+0000","last_peered":"2023-11-16T23:58:51.975325+0000","last_clean":"2023-11-16T23:58:51.975325+0000","last_became_active":"2023-11-16T23:58:46.288746+0000","last_became_peered":"2023-11-16T23:58:46.288746+0000","last_unstale":"2023-11-16T23:58:51.975325+0000","last_undegraded":"2023-11-16T23:58:51.975325+0000","last_fullsized":"2023-11-16T23:58:51.975325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:39:36.326325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239625+0000","last_change":"2023-11-16T23:58:46.288862+0000","last_active":"2023-11-16T23:58:51.239625+0000","last_peered":"2023-11-16T23:58:51.239625+0000","last_clean":"2023-11-16T23:58:51.239625+0000","last_became_active":"2023-11-16T23:58:46.288046+0000","last_became_peered":"2023-11-16T23:58:46.288046+0000","last_unstale":"2023-11-16T23:58:51.239625+0000","last_undegraded":"2023-11-16T23:58:51.239625+0000","last_fullsized":"2023-11-16T23:58:51.239625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:28:13.350271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974869+0000","last_change":"2023-11-16T23:58:46.283219+0000","last_active":"2023-11-16T23:58:51.974869+0000","last_peered":"2023-11-16T23:58:51.974869+0000","last_clean":"2023-11-16T23:58:51.974869+0000","last_became_active":"2023-11-16T23:58:46.282759+0000","last_became_peered":"2023-11-16T23:58:46.282759+0000","last_unstale":"2023-11-16T23:58:51.974869+0000","last_undegraded":"2023-11-16T23:58:51.974869+0000","last_fullsized":"2023-11-16T23:58:51.974869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:02:48.010586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:46.292139+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:46.291682+0000","last_became_peered":"2023-11-16T23:58:46.291682+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:27:16.238637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242504+0000","last_change":"2023-11-16T23:58:46.293326+0000","last_active":"2023-11-16T23:58:51.242504+0000","last_peered":"2023-11-16T23:58:51.242504+0000","last_clean":"2023-11-16T23:58:51.242504+0000","last_became_active":"2023-11-16T23:58:46.286619+0000","last_became_peered":"2023-11-16T23:58:46.286619+0000","last_unstale":"2023-11-16T23:58:51.242504+0000","last_undegraded":"2023-11-16T23:58:51.242504+0000","last_fullsized":"2023-11-16T23:58:51.242504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:41.213234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242574+0000","last_change":"2023-11-16T23:58:46.274095+0000","last_active":"2023-11-16T23:58:51.242574+0000","last_peered":"2023-11-16T23:58:51.242574+0000","last_clean":"2023-11-16T23:58:51.242574+0000","last_became_active":"2023-11-16T23:58:46.272718+0000","last_became_peered":"2023-11-16T23:58:46.272718+0000","last_unstale":"2023-11-16T23:58:51.242574+0000","last_undegraded":"2023-11-16T23:58:51.242574+0000","last_fullsized":"2023-11-16T23:58:51.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:45.267014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973636+0000","last_change":"2023-11-16T23:58:46.294079+0000","last_active":"2023-11-16T23:58:51.973636+0000","last_peered":"2023-11-16T23:58:51.973636+0000","last_clean":"2023-11-16T23:58:51.973636+0000","last_became_active":"2023-11-16T23:58:46.293738+0000","last_became_peered":"2023-11-16T23:58:46.293738+0000","last_unstale":"2023-11-16T23:58:51.973636+0000","last_undegraded":"2023-11-16T23:58:51.973636+0000","last_fullsized":"2023-11-16T23:58:51.973636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:48.113990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241160+0000","last_change":"2023-11-16T23:58:46.296439+0000","last_active":"2023-11-16T23:58:51.241160+0000","last_peered":"2023-11-16T23:58:51.241160+0000","last_clean":"2023-11-16T23:58:51.241160+0000","last_became_active":"2023-11-16T23:58:46.296002+0000","last_became_peered":"2023-11-16T23:58:46.296002+0000","last_unstale":"2023-11-16T23:58:51.241160+0000","last_undegraded":"2023-11-16T23:58:51.241160+0000","last_fullsized":"2023-11-16T23:58:51.241160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:45.679213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975910+0000","last_change":"2023-11-16T23:58:46.287374+0000","last_active":"2023-11-16T23:58:51.975910+0000","last_peered":"2023-11-16T23:58:51.975910+0000","last_clean":"2023-11-16T23:58:51.975910+0000","last_became_active":"2023-11-16T23:58:46.287044+0000","last_became_peered":"2023-11-16T23:58:46.287044+0000","last_unstale":"2023-11-16T23:58:51.975910+0000","last_undegraded":"2023-11-16T23:58:51.975910+0000","last_fullsized":"2023-11-16T23:58:51.975910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:31:41.509204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416818+0000","last_change":"2023-11-16T23:58:46.284627+0000","last_active":"2023-11-16T23:58:51.416818+0000","last_peered":"2023-11-16T23:58:51.416818+0000","last_clean":"2023-11-16T23:58:51.416818+0000","last_became_active":"2023-11-16T23:58:46.284221+0000","last_became_peered":"2023-11-16T23:58:46.284221+0000","last_unstale":"2023-11-16T23:58:51.416818+0000","last_undegraded":"2023-11-16T23:58:51.416818+0000","last_fullsized":"2023-11-16T23:58:51.416818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:37.717489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416711+0000","last_change":"2023-11-16T23:58:50.742742+0000","last_active":"2023-11-16T23:58:51.416711+0000","last_peered":"2023-11-16T23:58:51.416711+0000","last_clean":"2023-11-16T23:58:51.416711+0000","last_became_active":"2023-11-16T23:58:50.742337+0000","last_became_peered":"2023-11-16T23:58:50.742337+0000","last_unstale":"2023-11-16T23:58:51.416711+0000","last_undegraded":"2023-11-16T23:58:51.416711+0000","last_fullsized":"2023-11-16T23:58:51.416711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:03:40.742862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717374+0000","last_change":"2023-11-16T23:58:45.714612+0000","last_active":"2023-11-16T23:59:33.717374+0000","last_peered":"2023-11-16T23:59:33.717374+0000","last_clean":"2023-11-16T23:59:33.717374+0000","last_became_active":"2023-11-16T23:58:45.714235+0000","last_became_peered":"2023-11-16T23:58:45.714235+0000","last_unstale":"2023-11-16T23:59:33.717374+0000","last_undegraded":"2023-11-16T23:59:33.717374+0000","last_fullsized":"2023-11-16T23:59:33.717374+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:20:19.287101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974930+0000","last_change":"2023-11-16T23:58:46.294399+0000","last_active":"2023-11-16T23:58:51.974930+0000","last_peered":"2023-11-16T23:58:51.974930+0000","last_clean":"2023-11-16T23:58:51.974930+0000","last_became_active":"2023-11-16T23:58:46.291722+0000","last_became_peered":"2023-11-16T23:58:46.291722+0000","last_unstale":"2023-11-16T23:58:51.974930+0000","last_undegraded":"2023-11-16T23:58:51.974930+0000","last_fullsized":"2023-11-16T23:58:51.974930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:58:25.110448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973733+0000","last_change":"2023-11-16T23:58:50.745764+0000","last_active":"2023-11-16T23:58:51.973733+0000","last_peered":"2023-11-16T23:58:51.973733+0000","last_clean":"2023-11-16T23:58:51.973733+0000","last_became_active":"2023-11-16T23:58:50.745411+0000","last_became_peered":"2023-11-16T23:58:50.745411+0000","last_unstale":"2023-11-16T23:58:51.973733+0000","last_undegraded":"2023-11-16T23:58:51.973733+0000","last_fullsized":"2023-11-16T23:58:51.973733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:04:04.552603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975096+0000","last_change":"2023-11-16T23:58:45.724215+0000","last_active":"2023-11-16T23:58:51.975096+0000","last_peered":"2023-11-16T23:58:51.975096+0000","last_clean":"2023-11-16T23:58:51.975096+0000","last_became_active":"2023-11-16T23:58:45.723866+0000","last_became_peered":"2023-11-16T23:58:45.723866+0000","last_unstale":"2023-11-16T23:58:51.975096+0000","last_undegraded":"2023-11-16T23:58:51.975096+0000","last_fullsized":"2023-11-16T23:58:51.975096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:09.212226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243119+0000","last_change":"2023-11-16T23:58:46.291701+0000","last_active":"2023-11-16T23:58:51.243119+0000","last_peered":"2023-11-16T23:58:51.243119+0000","last_clean":"2023-11-16T23:58:51.243119+0000","last_became_active":"2023-11-16T23:58:46.282612+0000","last_became_peered":"2023-11-16T23:58:46.282612+0000","last_unstale":"2023-11-16T23:58:51.243119+0000","last_undegraded":"2023-11-16T23:58:51.243119+0000","last_fullsized":"2023-11-16T23:58:51.243119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:18.169006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975347+0000","last_change":"2023-11-16T23:58:50.731025+0000","last_active":"2023-11-16T23:58:51.975347+0000","last_peered":"2023-11-16T23:58:51.975347+0000","last_clean":"2023-11-16T23:58:51.975347+0000","last_became_active":"2023-11-16T23:58:50.730593+0000","last_became_peered":"2023-11-16T23:58:50.730593+0000","last_unstale":"2023-11-16T23:58:51.975347+0000","last_undegraded":"2023-11-16T23:58:51.975347+0000","last_fullsized":"2023-11-16T23:58:51.975347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:18.213726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715505+0000","last_change":"2023-11-16T23:58:45.716403+0000","last_active":"2023-11-16T23:59:33.715505+0000","last_peered":"2023-11-16T23:59:33.715505+0000","last_clean":"2023-11-16T23:59:33.715505+0000","last_became_active":"2023-11-16T23:58:45.716050+0000","last_became_peered":"2023-11-16T23:58:45.716050+0000","last_unstale":"2023-11-16T23:59:33.715505+0000","last_undegraded":"2023-11-16T23:59:33.715505+0000","last_fullsized":"2023-11-16T23:59:33.715505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:42:55.310961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974544+0000","last_change":"2023-11-16T23:58:46.286460+0000","last_active":"2023-11-16T23:58:51.974544+0000","last_peered":"2023-11-16T23:58:51.974544+0000","last_clean":"2023-11-16T23:58:51.974544+0000","last_became_active":"2023-11-16T23:58:46.286135+0000","last_became_peered":"2023-11-16T23:58:46.286135+0000","last_unstale":"2023-11-16T23:58:51.974544+0000","last_undegraded":"2023-11-16T23:58:51.974544+0000","last_fullsized":"2023-11-16T23:58:51.974544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:15:02.309346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973789+0000","last_change":"2023-11-16T23:58:50.737134+0000","last_active":"2023-11-16T23:58:51.973789+0000","last_peered":"2023-11-16T23:58:51.973789+0000","last_clean":"2023-11-16T23:58:51.973789+0000","last_became_active":"2023-11-16T23:58:50.736489+0000","last_became_peered":"2023-11-16T23:58:50.736489+0000","last_unstale":"2023-11-16T23:58:51.973789+0000","last_undegraded":"2023-11-16T23:58:51.973789+0000","last_fullsized":"2023-11-16T23:58:51.973789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:33.961942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973818+0000","last_change":"2023-11-16T23:58:45.713153+0000","last_active":"2023-11-16T23:58:51.973818+0000","last_peered":"2023-11-16T23:58:51.973818+0000","last_clean":"2023-11-16T23:58:51.973818+0000","last_became_active":"2023-11-16T23:58:45.712626+0000","last_became_peered":"2023-11-16T23:58:45.712626+0000","last_unstale":"2023-11-16T23:58:51.973818+0000","last_undegraded":"2023-11-16T23:58:51.973818+0000","last_fullsized":"2023-11-16T23:58:51.973818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:55:36.907838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584145+0000","last_change":"2023-11-16T23:58:46.289729+0000","last_active":"2023-11-16T23:58:51.584145+0000","last_peered":"2023-11-16T23:58:51.584145+0000","last_clean":"2023-11-16T23:58:51.584145+0000","last_became_active":"2023-11-16T23:58:46.284791+0000","last_became_peered":"2023-11-16T23:58:46.284791+0000","last_unstale":"2023-11-16T23:58:51.584145+0000","last_undegraded":"2023-11-16T23:58:51.584145+0000","last_fullsized":"2023-11-16T23:58:51.584145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:13:27.839983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416962+0000","last_change":"2023-11-16T23:58:50.725585+0000","last_active":"2023-11-16T23:58:51.416962+0000","last_peered":"2023-11-16T23:58:51.416962+0000","last_clean":"2023-11-16T23:58:51.416962+0000","last_became_active":"2023-11-16T23:58:50.725188+0000","last_became_peered":"2023-11-16T23:58:50.725188+0000","last_unstale":"2023-11-16T23:58:51.416962+0000","last_undegraded":"2023-11-16T23:58:51.416962+0000","last_fullsized":"2023-11-16T23:58:51.416962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:54.017294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:16.772654+0000","last_change":"2023-11-16T23:58:45.715656+0000","last_active":"2023-11-17T00:00:16.772654+0000","last_peered":"2023-11-17T00:00:16.772654+0000","last_clean":"2023-11-17T00:00:16.772654+0000","last_became_active":"2023-11-16T23:58:45.715331+0000","last_became_peered":"2023-11-16T23:58:45.715331+0000","last_unstale":"2023-11-17T00:00:16.772654+0000","last_undegraded":"2023-11-17T00:00:16.772654+0000","last_fullsized":"2023-11-17T00:00:16.772654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:38:28.590646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973481+0000","last_change":"2023-11-16T23:58:46.294442+0000","last_active":"2023-11-16T23:58:51.973481+0000","last_peered":"2023-11-16T23:58:51.973481+0000","last_clean":"2023-11-16T23:58:51.973481+0000","last_became_active":"2023-11-16T23:58:46.294112+0000","last_became_peered":"2023-11-16T23:58:46.294112+0000","last_unstale":"2023-11-16T23:58:51.973481+0000","last_undegraded":"2023-11-16T23:58:51.973481+0000","last_fullsized":"2023-11-16T23:58:51.973481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:16:37.966411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416678+0000","last_change":"2023-11-16T23:58:50.747104+0000","last_active":"2023-11-16T23:58:51.416678+0000","last_peered":"2023-11-16T23:58:51.416678+0000","last_clean":"2023-11-16T23:58:51.416678+0000","last_became_active":"2023-11-16T23:58:50.746684+0000","last_became_peered":"2023-11-16T23:58:50.746684+0000","last_unstale":"2023-11-16T23:58:51.416678+0000","last_undegraded":"2023-11-16T23:58:51.416678+0000","last_fullsized":"2023-11-16T23:58:51.416678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:25.519731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715891+0000","last_change":"2023-11-16T23:58:45.719225+0000","last_active":"2023-11-16T23:59:33.715891+0000","last_peered":"2023-11-16T23:59:33.715891+0000","last_clean":"2023-11-16T23:59:33.715891+0000","last_became_active":"2023-11-16T23:58:45.718815+0000","last_became_peered":"2023-11-16T23:58:45.718815+0000","last_unstale":"2023-11-16T23:59:33.715891+0000","last_undegraded":"2023-11-16T23:59:33.715891+0000","last_fullsized":"2023-11-16T23:59:33.715891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:09.302556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583967+0000","last_change":"2023-11-16T23:58:46.292334+0000","last_active":"2023-11-16T23:58:51.583967+0000","last_peered":"2023-11-16T23:58:51.583967+0000","last_clean":"2023-11-16T23:58:51.583967+0000","last_became_active":"2023-11-16T23:58:46.291882+0000","last_became_peered":"2023-11-16T23:58:46.291882+0000","last_unstale":"2023-11-16T23:58:51.583967+0000","last_undegraded":"2023-11-16T23:58:51.583967+0000","last_fullsized":"2023-11-16T23:58:51.583967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:17.751014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241151+0000","last_change":"2023-11-16T23:58:50.742737+0000","last_active":"2023-11-16T23:58:51.241151+0000","last_peered":"2023-11-16T23:58:51.241151+0000","last_clean":"2023-11-16T23:58:51.241151+0000","last_became_active":"2023-11-16T23:58:50.742257+0000","last_became_peered":"2023-11-16T23:58:50.742257+0000","last_unstale":"2023-11-16T23:58:51.241151+0000","last_undegraded":"2023-11-16T23:58:51.241151+0000","last_fullsized":"2023-11-16T23:58:51.241151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:47:46.061379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715391+0000","last_change":"2023-11-16T23:58:45.716537+0000","last_active":"2023-11-16T23:59:33.715391+0000","last_peered":"2023-11-16T23:59:33.715391+0000","last_clean":"2023-11-16T23:59:33.715391+0000","last_became_active":"2023-11-16T23:58:45.716018+0000","last_became_peered":"2023-11-16T23:58:45.716018+0000","last_unstale":"2023-11-16T23:59:33.715391+0000","last_undegraded":"2023-11-16T23:59:33.715391+0000","last_fullsized":"2023-11-16T23:59:33.715391+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:56:57.891968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973828+0000","last_change":"2023-11-16T23:58:46.288713+0000","last_active":"2023-11-16T23:58:51.973828+0000","last_peered":"2023-11-16T23:58:51.973828+0000","last_clean":"2023-11-16T23:58:51.973828+0000","last_became_active":"2023-11-16T23:58:46.288363+0000","last_became_peered":"2023-11-16T23:58:46.288363+0000","last_unstale":"2023-11-16T23:58:51.973828+0000","last_undegraded":"2023-11-16T23:58:51.973828+0000","last_fullsized":"2023-11-16T23:58:51.973828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:48:49.725514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243013+0000","last_change":"2023-11-16T23:58:50.728730+0000","last_active":"2023-11-16T23:58:51.243013+0000","last_peered":"2023-11-16T23:58:51.243013+0000","last_clean":"2023-11-16T23:58:51.243013+0000","last_became_active":"2023-11-16T23:58:50.728164+0000","last_became_peered":"2023-11-16T23:58:50.728164+0000","last_unstale":"2023-11-16T23:58:51.243013+0000","last_undegraded":"2023-11-16T23:58:51.243013+0000","last_fullsized":"2023-11-16T23:58:51.243013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:30:15.323131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583464+0000","last_change":"2023-11-16T23:58:45.713056+0000","last_active":"2023-11-16T23:58:51.583464+0000","last_peered":"2023-11-16T23:58:51.583464+0000","last_clean":"2023-11-16T23:58:51.583464+0000","last_became_active":"2023-11-16T23:58:45.712685+0000","last_became_peered":"2023-11-16T23:58:45.712685+0000","last_unstale":"2023-11-16T23:58:51.583464+0000","last_undegraded":"2023-11-16T23:58:51.583464+0000","last_fullsized":"2023-11-16T23:58:51.583464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:35.811313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974076+0000","last_change":"2023-11-16T23:58:46.291540+0000","last_active":"2023-11-16T23:58:51.974076+0000","last_peered":"2023-11-16T23:58:51.974076+0000","last_clean":"2023-11-16T23:58:51.974076+0000","last_became_active":"2023-11-16T23:58:46.289372+0000","last_became_peered":"2023-11-16T23:58:46.289372+0000","last_unstale":"2023-11-16T23:58:51.974076+0000","last_undegraded":"2023-11-16T23:58:51.974076+0000","last_fullsized":"2023-11-16T23:58:51.974076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:38:19.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974448+0000","last_change":"2023-11-16T23:58:50.729362+0000","last_active":"2023-11-16T23:58:51.974448+0000","last_peered":"2023-11-16T23:58:51.974448+0000","last_clean":"2023-11-16T23:58:51.974448+0000","last_became_active":"2023-11-16T23:58:50.729007+0000","last_became_peered":"2023-11-16T23:58:50.729007+0000","last_unstale":"2023-11-16T23:58:51.974448+0000","last_undegraded":"2023-11-16T23:58:51.974448+0000","last_fullsized":"2023-11-16T23:58:51.974448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:59.110409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:17.773533+0000","last_change":"2023-11-16T23:58:45.717814+0000","last_active":"2023-11-17T00:00:17.773533+0000","last_peered":"2023-11-17T00:00:17.773533+0000","last_clean":"2023-11-17T00:00:17.773533+0000","last_became_active":"2023-11-16T23:58:45.717494+0000","last_became_peered":"2023-11-16T23:58:45.717494+0000","last_unstale":"2023-11-17T00:00:17.773533+0000","last_undegraded":"2023-11-17T00:00:17.773533+0000","last_fullsized":"2023-11-17T00:00:17.773533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:33:17.157376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974442+0000","last_change":"2023-11-16T23:58:46.285305+0000","last_active":"2023-11-16T23:58:51.974442+0000","last_peered":"2023-11-16T23:58:51.974442+0000","last_clean":"2023-11-16T23:58:51.974442+0000","last_became_active":"2023-11-16T23:58:46.284563+0000","last_became_peered":"2023-11-16T23:58:46.284563+0000","last_unstale":"2023-11-16T23:58:51.974442+0000","last_undegraded":"2023-11-16T23:58:51.974442+0000","last_fullsized":"2023-11-16T23:58:51.974442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:12.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973969+0000","last_change":"2023-11-16T23:58:50.740863+0000","last_active":"2023-11-16T23:58:51.973969+0000","last_peered":"2023-11-16T23:58:51.973969+0000","last_clean":"2023-11-16T23:58:51.973969+0000","last_became_active":"2023-11-16T23:58:50.740400+0000","last_became_peered":"2023-11-16T23:58:50.740400+0000","last_unstale":"2023-11-16T23:58:51.973969+0000","last_undegraded":"2023-11-16T23:58:51.973969+0000","last_fullsized":"2023-11-16T23:58:51.973969+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:52:44.691356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733800+0000","last_change":"2023-11-16T23:58:45.715701+0000","last_active":"2023-11-16T23:59:32.733800+0000","last_peered":"2023-11-16T23:59:32.733800+0000","last_clean":"2023-11-16T23:59:32.733800+0000","last_became_active":"2023-11-16T23:58:45.715273+0000","last_became_peered":"2023-11-16T23:58:45.715273+0000","last_unstale":"2023-11-16T23:59:32.733800+0000","last_undegraded":"2023-11-16T23:59:32.733800+0000","last_fullsized":"2023-11-16T23:59:32.733800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:08.019018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243105+0000","last_change":"2023-11-16T23:58:46.294740+0000","last_active":"2023-11-16T23:58:51.243105+0000","last_peered":"2023-11-16T23:58:51.243105+0000","last_clean":"2023-11-16T23:58:51.243105+0000","last_became_active":"2023-11-16T23:58:46.282681+0000","last_became_peered":"2023-11-16T23:58:46.282681+0000","last_unstale":"2023-11-16T23:58:51.243105+0000","last_undegraded":"2023-11-16T23:58:51.243105+0000","last_fullsized":"2023-11-16T23:58:51.243105+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:44.418038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975559+0000","last_change":"2023-11-16T23:58:50.730244+0000","last_active":"2023-11-16T23:58:51.975559+0000","last_peered":"2023-11-16T23:58:51.975559+0000","last_clean":"2023-11-16T23:58:51.975559+0000","last_became_active":"2023-11-16T23:58:50.729669+0000","last_became_peered":"2023-11-16T23:58:50.729669+0000","last_unstale":"2023-11-16T23:58:51.975559+0000","last_undegraded":"2023-11-16T23:58:51.975559+0000","last_fullsized":"2023-11-16T23:58:51.975559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:47:57.416754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733951+0000","last_change":"2023-11-16T23:58:45.722939+0000","last_active":"2023-11-16T23:59:32.733951+0000","last_peered":"2023-11-16T23:59:32.733951+0000","last_clean":"2023-11-16T23:59:32.733951+0000","last_became_active":"2023-11-16T23:58:45.722559+0000","last_became_peered":"2023-11-16T23:58:45.722559+0000","last_unstale":"2023-11-16T23:59:32.733951+0000","last_undegraded":"2023-11-16T23:59:32.733951+0000","last_fullsized":"2023-11-16T23:59:32.733951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:38.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974381+0000","last_change":"2023-11-16T23:58:46.285249+0000","last_active":"2023-11-16T23:58:51.974381+0000","last_peered":"2023-11-16T23:58:51.974381+0000","last_clean":"2023-11-16T23:58:51.974381+0000","last_became_active":"2023-11-16T23:58:46.284815+0000","last_became_peered":"2023-11-16T23:58:46.284815+0000","last_unstale":"2023-11-16T23:58:51.974381+0000","last_undegraded":"2023-11-16T23:58:51.974381+0000","last_fullsized":"2023-11-16T23:58:51.974381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:28.991224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974731+0000","last_change":"2023-11-16T23:58:50.734889+0000","last_active":"2023-11-16T23:58:51.974731+0000","last_peered":"2023-11-16T23:58:51.974731+0000","last_clean":"2023-11-16T23:58:51.974731+0000","last_became_active":"2023-11-16T23:58:50.734461+0000","last_became_peered":"2023-11-16T23:58:50.734461+0000","last_unstale":"2023-11-16T23:58:51.974731+0000","last_undegraded":"2023-11-16T23:58:51.974731+0000","last_fullsized":"2023-11-16T23:58:51.974731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:34:05.742581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":155,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:18.780724+0000","last_change":"2023-11-16T23:58:45.716428+0000","last_active":"2023-11-17T00:00:18.780724+0000","last_peered":"2023-11-17T00:00:18.780724+0000","last_clean":"2023-11-17T00:00:18.780724+0000","last_became_active":"2023-11-16T23:58:45.716076+0000","last_became_peered":"2023-11-16T23:58:45.716076+0000","last_unstale":"2023-11-17T00:00:18.780724+0000","last_undegraded":"2023-11-17T00:00:18.780724+0000","last_fullsized":"2023-11-17T00:00:18.780724+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:14:06.840585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60393,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":108,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973844+0000","last_change":"2023-11-16T23:58:46.282889+0000","last_active":"2023-11-16T23:58:51.973844+0000","last_peered":"2023-11-16T23:58:51.973844+0000","last_clean":"2023-11-16T23:58:51.973844+0000","last_became_active":"2023-11-16T23:58:46.282450+0000","last_became_peered":"2023-11-16T23:58:46.282450+0000","last_unstale":"2023-11-16T23:58:51.973844+0000","last_undegraded":"2023-11-16T23:58:51.973844+0000","last_fullsized":"2023-11-16T23:58:51.973844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:08:09.378619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584460+0000","last_change":"2023-11-16T23:58:50.735438+0000","last_active":"2023-11-16T23:58:51.584460+0000","last_peered":"2023-11-16T23:58:51.584460+0000","last_clean":"2023-11-16T23:58:51.584460+0000","last_became_active":"2023-11-16T23:58:50.735040+0000","last_became_peered":"2023-11-16T23:58:50.735040+0000","last_unstale":"2023-11-16T23:58:51.584460+0000","last_undegraded":"2023-11-16T23:58:51.584460+0000","last_fullsized":"2023-11-16T23:58:51.584460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:20.496063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.716638+0000","last_change":"2023-11-16T23:58:45.721161+0000","last_active":"2023-11-16T23:59:33.716638+0000","last_peered":"2023-11-16T23:59:33.716638+0000","last_clean":"2023-11-16T23:59:33.716638+0000","last_became_active":"2023-11-16T23:58:45.720826+0000","last_became_peered":"2023-11-16T23:58:45.720826+0000","last_unstale":"2023-11-16T23:59:33.716638+0000","last_undegraded":"2023-11-16T23:59:33.716638+0000","last_fullsized":"2023-11-16T23:59:33.716638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:33.101318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974461+0000","last_change":"2023-11-16T23:58:46.293152+0000","last_active":"2023-11-16T23:58:51.974461+0000","last_peered":"2023-11-16T23:58:51.974461+0000","last_clean":"2023-11-16T23:58:51.974461+0000","last_became_active":"2023-11-16T23:58:46.292830+0000","last_became_peered":"2023-11-16T23:58:46.292830+0000","last_unstale":"2023-11-16T23:58:51.974461+0000","last_undegraded":"2023-11-16T23:58:51.974461+0000","last_fullsized":"2023-11-16T23:58:51.974461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:06.803488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240824+0000","last_change":"2023-11-16T23:58:50.739327+0000","last_active":"2023-11-16T23:58:51.240824+0000","last_peered":"2023-11-16T23:58:51.240824+0000","last_clean":"2023-11-16T23:58:51.240824+0000","last_became_active":"2023-11-16T23:58:50.739024+0000","last_became_peered":"2023-11-16T23:58:50.739024+0000","last_unstale":"2023-11-16T23:58:51.240824+0000","last_undegraded":"2023-11-16T23:58:51.240824+0000","last_fullsized":"2023-11-16T23:58:51.240824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:58.095409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734178+0000","last_change":"2023-11-16T23:58:45.716730+0000","last_active":"2023-11-16T23:59:32.734178+0000","last_peered":"2023-11-16T23:59:32.734178+0000","last_clean":"2023-11-16T23:59:32.734178+0000","last_became_active":"2023-11-16T23:58:45.716380+0000","last_became_peered":"2023-11-16T23:58:45.716380+0000","last_unstale":"2023-11-16T23:59:32.734178+0000","last_undegraded":"2023-11-16T23:59:32.734178+0000","last_fullsized":"2023-11-16T23:59:32.734178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:06:26.975787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239189+0000","last_change":"2023-11-16T23:58:46.290867+0000","last_active":"2023-11-16T23:58:51.239189+0000","last_peered":"2023-11-16T23:58:51.239189+0000","last_clean":"2023-11-16T23:58:51.239189+0000","last_became_active":"2023-11-16T23:58:46.290464+0000","last_became_peered":"2023-11-16T23:58:46.290464+0000","last_unstale":"2023-11-16T23:58:51.239189+0000","last_undegraded":"2023-11-16T23:58:51.239189+0000","last_fullsized":"2023-11-16T23:58:51.239189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:23.908610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975901+0000","last_change":"2023-11-16T23:58:50.747045+0000","last_active":"2023-11-16T23:58:51.975901+0000","last_peered":"2023-11-16T23:58:51.975901+0000","last_clean":"2023-11-16T23:58:51.975901+0000","last_became_active":"2023-11-16T23:58:50.743955+0000","last_became_peered":"2023-11-16T23:58:50.743955+0000","last_unstale":"2023-11-16T23:58:51.975901+0000","last_undegraded":"2023-11-16T23:58:51.975901+0000","last_fullsized":"2023-11-16T23:58:51.975901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:54:18.403005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:14.775730+0000","last_change":"2023-11-16T23:58:45.714767+0000","last_active":"2023-11-17T00:00:14.775730+0000","last_peered":"2023-11-17T00:00:14.775730+0000","last_clean":"2023-11-17T00:00:14.775730+0000","last_became_active":"2023-11-16T23:58:45.713817+0000","last_became_peered":"2023-11-16T23:58:45.713817+0000","last_unstale":"2023-11-17T00:00:14.775730+0000","last_undegraded":"2023-11-17T00:00:14.775730+0000","last_fullsized":"2023-11-17T00:00:14.775730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:44:48.694382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238988+0000","last_change":"2023-11-16T23:58:46.295593+0000","last_active":"2023-11-16T23:58:51.238988+0000","last_peered":"2023-11-16T23:58:51.238988+0000","last_clean":"2023-11-16T23:58:51.238988+0000","last_became_active":"2023-11-16T23:58:46.295253+0000","last_became_peered":"2023-11-16T23:58:46.295253+0000","last_unstale":"2023-11-16T23:58:51.238988+0000","last_undegraded":"2023-11-16T23:58:51.238988+0000","last_fullsized":"2023-11-16T23:58:51.238988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:57:11.390297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976616+0000","last_change":"2023-11-16T23:58:50.730370+0000","last_active":"2023-11-16T23:58:51.976616+0000","last_peered":"2023-11-16T23:58:51.976616+0000","last_clean":"2023-11-16T23:58:51.976616+0000","last_became_active":"2023-11-16T23:58:50.729915+0000","last_became_peered":"2023-11-16T23:58:50.729915+0000","last_unstale":"2023-11-16T23:58:51.976616+0000","last_undegraded":"2023-11-16T23:58:51.976616+0000","last_fullsized":"2023-11-16T23:58:51.976616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:40:19.878796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718774+0000","last_change":"2023-11-16T23:58:45.715792+0000","last_active":"2023-11-16T23:59:33.718774+0000","last_peered":"2023-11-16T23:59:33.718774+0000","last_clean":"2023-11-16T23:59:33.718774+0000","last_became_active":"2023-11-16T23:58:45.715454+0000","last_became_peered":"2023-11-16T23:58:45.715454+0000","last_unstale":"2023-11-16T23:59:33.718774+0000","last_undegraded":"2023-11-16T23:59:33.718774+0000","last_fullsized":"2023-11-16T23:59:33.718774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:51.440176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583583+0000","last_change":"2023-11-16T23:58:46.288249+0000","last_active":"2023-11-16T23:58:51.583583+0000","last_peered":"2023-11-16T23:58:51.583583+0000","last_clean":"2023-11-16T23:58:51.583583+0000","last_became_active":"2023-11-16T23:58:46.287916+0000","last_became_peered":"2023-11-16T23:58:46.287916+0000","last_unstale":"2023-11-16T23:58:51.583583+0000","last_undegraded":"2023-11-16T23:58:51.583583+0000","last_fullsized":"2023-11-16T23:58:51.583583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:34:19.800012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583795+0000","last_change":"2023-11-16T23:58:50.733835+0000","last_active":"2023-11-16T23:58:51.583795+0000","last_peered":"2023-11-16T23:58:51.583795+0000","last_clean":"2023-11-16T23:58:51.583795+0000","last_became_active":"2023-11-16T23:58:50.733373+0000","last_became_peered":"2023-11-16T23:58:50.733373+0000","last_unstale":"2023-11-16T23:58:51.583795+0000","last_undegraded":"2023-11-16T23:58:51.583795+0000","last_fullsized":"2023-11-16T23:58:51.583795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:27:17.246864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.704975+0000","last_change":"2023-11-16T23:58:45.712055+0000","last_active":"2023-11-16T23:59:33.704975+0000","last_peered":"2023-11-16T23:59:33.704975+0000","last_clean":"2023-11-16T23:59:33.704975+0000","last_became_active":"2023-11-16T23:58:45.711618+0000","last_became_peered":"2023-11-16T23:58:45.711618+0000","last_unstale":"2023-11-16T23:59:33.704975+0000","last_undegraded":"2023-11-16T23:59:33.704975+0000","last_fullsized":"2023-11-16T23:59:33.704975+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:37:41.700022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241050+0000","last_change":"2023-11-16T23:58:46.291460+0000","last_active":"2023-11-16T23:58:51.241050+0000","last_peered":"2023-11-16T23:58:51.241050+0000","last_clean":"2023-11-16T23:58:51.241050+0000","last_became_active":"2023-11-16T23:58:46.291144+0000","last_became_peered":"2023-11-16T23:58:46.291144+0000","last_unstale":"2023-11-16T23:58:51.241050+0000","last_undegraded":"2023-11-16T23:58:51.241050+0000","last_fullsized":"2023-11-16T23:58:51.241050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:24:10.888038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973559+0000","last_change":"2023-11-16T23:58:50.927320+0000","last_active":"2023-11-16T23:58:51.973559+0000","last_peered":"2023-11-16T23:58:51.973559+0000","last_clean":"2023-11-16T23:58:51.973559+0000","last_became_active":"2023-11-16T23:58:50.926767+0000","last_became_peered":"2023-11-16T23:58:50.926767+0000","last_unstale":"2023-11-16T23:58:51.973559+0000","last_undegraded":"2023-11-16T23:58:51.973559+0000","last_fullsized":"2023-11-16T23:58:51.973559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:21.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975281+0000","last_change":"2023-11-16T23:58:45.715654+0000","last_active":"2023-11-16T23:58:51.975281+0000","last_peered":"2023-11-16T23:58:51.975281+0000","last_clean":"2023-11-16T23:58:51.975281+0000","last_became_active":"2023-11-16T23:58:45.714899+0000","last_became_peered":"2023-11-16T23:58:45.714899+0000","last_unstale":"2023-11-16T23:58:51.975281+0000","last_undegraded":"2023-11-16T23:58:51.975281+0000","last_fullsized":"2023-11-16T23:58:51.975281+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:53.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974886+0000","last_change":"2023-11-16T23:58:46.291818+0000","last_active":"2023-11-16T23:58:51.974886+0000","last_peered":"2023-11-16T23:58:51.974886+0000","last_clean":"2023-11-16T23:58:51.974886+0000","last_became_active":"2023-11-16T23:58:46.291517+0000","last_became_peered":"2023-11-16T23:58:46.291517+0000","last_unstale":"2023-11-16T23:58:51.974886+0000","last_undegraded":"2023-11-16T23:58:51.974886+0000","last_fullsized":"2023-11-16T23:58:51.974886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:51:34.124036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:50.733546+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:50.733009+0000","last_became_peered":"2023-11-16T23:58:50.733009+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:37:16.925270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239226+0000","last_change":"2023-11-16T23:58:45.719181+0000","last_active":"2023-11-16T23:58:51.239226+0000","last_peered":"2023-11-16T23:58:51.239226+0000","last_clean":"2023-11-16T23:58:51.239226+0000","last_became_active":"2023-11-16T23:58:45.718826+0000","last_became_peered":"2023-11-16T23:58:45.718826+0000","last_unstale":"2023-11-16T23:58:51.239226+0000","last_undegraded":"2023-11-16T23:58:51.239226+0000","last_fullsized":"2023-11-16T23:58:51.239226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:52:25.180163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975356+0000","last_change":"2023-11-16T23:58:46.283405+0000","last_active":"2023-11-16T23:58:51.975356+0000","last_peered":"2023-11-16T23:58:51.975356+0000","last_clean":"2023-11-16T23:58:51.975356+0000","last_became_active":"2023-11-16T23:58:46.282995+0000","last_became_peered":"2023-11-16T23:58:46.282995+0000","last_unstale":"2023-11-16T23:58:51.975356+0000","last_undegraded":"2023-11-16T23:58:51.975356+0000","last_fullsized":"2023-11-16T23:58:51.975356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:53:28.645169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976551+0000","last_change":"2023-11-16T23:58:50.928220+0000","last_active":"2023-11-16T23:58:51.976551+0000","last_peered":"2023-11-16T23:58:51.976551+0000","last_clean":"2023-11-16T23:58:51.976551+0000","last_became_active":"2023-11-16T23:58:50.927006+0000","last_became_peered":"2023-11-16T23:58:50.927006+0000","last_unstale":"2023-11-16T23:58:51.976551+0000","last_undegraded":"2023-11-16T23:58:51.976551+0000","last_fullsized":"2023-11-16T23:58:51.976551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:09:21.135371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583555+0000","last_change":"2023-11-16T23:58:46.289260+0000","last_active":"2023-11-16T23:58:51.583555+0000","last_peered":"2023-11-16T23:58:51.583555+0000","last_clean":"2023-11-16T23:58:51.583555+0000","last_became_active":"2023-11-16T23:58:46.285041+0000","last_became_peered":"2023-11-16T23:58:46.285041+0000","last_unstale":"2023-11-16T23:58:51.583555+0000","last_undegraded":"2023-11-16T23:58:51.583555+0000","last_fullsized":"2023-11-16T23:58:51.583555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:16.675991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:17.779736+0000","last_change":"2023-11-16T23:58:45.726423+0000","last_active":"2023-11-17T00:00:17.779736+0000","last_peered":"2023-11-17T00:00:17.779736+0000","last_clean":"2023-11-17T00:00:17.779736+0000","last_became_active":"2023-11-16T23:58:45.725996+0000","last_became_peered":"2023-11-16T23:58:45.725996+0000","last_unstale":"2023-11-17T00:00:17.779736+0000","last_undegraded":"2023-11-17T00:00:17.779736+0000","last_fullsized":"2023-11-17T00:00:17.779736+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:38.058418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:52.683597+0000","last_change":"2023-11-16T23:58:50.730148+0000","last_active":"2023-11-16T23:58:52.683597+0000","last_peered":"2023-11-16T23:58:52.683597+0000","last_clean":"2023-11-16T23:58:52.683597+0000","last_became_active":"2023-11-16T23:58:50.729675+0000","last_became_peered":"2023-11-16T23:58:50.729675+0000","last_unstale":"2023-11-16T23:58:52.683597+0000","last_undegraded":"2023-11-16T23:58:52.683597+0000","last_fullsized":"2023-11-16T23:58:52.683597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:42:47.288350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974143+0000","last_change":"2023-11-16T23:58:46.295437+0000","last_active":"2023-11-16T23:58:51.974143+0000","last_peered":"2023-11-16T23:58:51.974143+0000","last_clean":"2023-11-16T23:58:51.974143+0000","last_became_active":"2023-11-16T23:58:46.291775+0000","last_became_peered":"2023-11-16T23:58:46.291775+0000","last_unstale":"2023-11-16T23:58:51.974143+0000","last_undegraded":"2023-11-16T23:58:51.974143+0000","last_fullsized":"2023-11-16T23:58:51.974143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:17.517488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714142+0000","last_change":"2023-11-16T23:58:45.714497+0000","last_active":"2023-11-16T23:59:33.714142+0000","last_peered":"2023-11-16T23:59:33.714142+0000","last_clean":"2023-11-16T23:59:33.714142+0000","last_became_active":"2023-11-16T23:58:45.714167+0000","last_became_peered":"2023-11-16T23:58:45.714167+0000","last_unstale":"2023-11-16T23:59:33.714142+0000","last_undegraded":"2023-11-16T23:59:33.714142+0000","last_fullsized":"2023-11-16T23:59:33.714142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:46.839077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238863+0000","last_change":"2023-11-16T23:57:37.700941+0000","last_active":"2023-11-16T23:58:51.238863+0000","last_peered":"2023-11-16T23:58:51.238863+0000","last_clean":"2023-11-16T23:58:51.238863+0000","last_became_active":"2023-11-16T23:57:37.700379+0000","last_became_peered":"2023-11-16T23:57:37.700379+0000","last_unstale":"2023-11-16T23:58:51.238863+0000","last_undegraded":"2023-11-16T23:58:51.238863+0000","last_fullsized":"2023-11-16T23:58:51.238863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_clean_scrub_stamp":"2023-11-16T23:54:03.597877+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:51:56.505655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584217+0000","last_change":"2023-11-16T23:58:50.927446+0000","last_active":"2023-11-16T23:58:51.584217+0000","last_peered":"2023-11-16T23:58:51.584217+0000","last_clean":"2023-11-16T23:58:51.584217+0000","last_became_active":"2023-11-16T23:58:50.927042+0000","last_became_peered":"2023-11-16T23:58:50.927042+0000","last_unstale":"2023-11-16T23:58:51.584217+0000","last_undegraded":"2023-11-16T23:58:51.584217+0000","last_fullsized":"2023-11-16T23:58:51.584217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:11:06.468189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973431+0000","last_change":"2023-11-16T23:58:46.283611+0000","last_active":"2023-11-16T23:58:51.973431+0000","last_peered":"2023-11-16T23:58:51.973431+0000","last_clean":"2023-11-16T23:58:51.973431+0000","last_became_active":"2023-11-16T23:58:46.281685+0000","last_became_peered":"2023-11-16T23:58:46.281685+0000","last_unstale":"2023-11-16T23:58:51.973431+0000","last_undegraded":"2023-11-16T23:58:51.973431+0000","last_fullsized":"2023-11-16T23:58:51.973431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:28:30.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714897+0000","last_change":"2023-11-16T23:58:45.716094+0000","last_active":"2023-11-16T23:59:33.714897+0000","last_peered":"2023-11-16T23:59:33.714897+0000","last_clean":"2023-11-16T23:59:33.714897+0000","last_became_active":"2023-11-16T23:58:45.715748+0000","last_became_peered":"2023-11-16T23:58:45.715748+0000","last_unstale":"2023-11-16T23:59:33.714897+0000","last_undegraded":"2023-11-16T23:59:33.714897+0000","last_fullsized":"2023-11-16T23:59:33.714897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:48.184073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240803+0000","last_change":"2023-11-16T23:58:50.737281+0000","last_active":"2023-11-16T23:58:51.240803+0000","last_peered":"2023-11-16T23:58:51.240803+0000","last_clean":"2023-11-16T23:58:51.240803+0000","last_became_active":"2023-11-16T23:58:50.736797+0000","last_became_peered":"2023-11-16T23:58:50.736797+0000","last_unstale":"2023-11-16T23:58:51.240803+0000","last_undegraded":"2023-11-16T23:58:51.240803+0000","last_fullsized":"2023-11-16T23:58:51.240803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:39.192922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734179+0000","last_change":"2023-11-16T23:58:45.722935+0000","last_active":"2023-11-16T23:59:32.734179+0000","last_peered":"2023-11-16T23:59:32.734179+0000","last_clean":"2023-11-16T23:59:32.734179+0000","last_became_active":"2023-11-16T23:58:45.722489+0000","last_became_peered":"2023-11-16T23:58:45.722489+0000","last_unstale":"2023-11-16T23:59:32.734179+0000","last_undegraded":"2023-11-16T23:59:32.734179+0000","last_fullsized":"2023-11-16T23:59:32.734179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:58.547593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975877+0000","last_change":"2023-11-16T23:58:46.284611+0000","last_active":"2023-11-16T23:58:51.975877+0000","last_peered":"2023-11-16T23:58:51.975877+0000","last_clean":"2023-11-16T23:58:51.975877+0000","last_became_active":"2023-11-16T23:58:46.283063+0000","last_became_peered":"2023-11-16T23:58:46.283063+0000","last_unstale":"2023-11-16T23:58:51.975877+0000","last_undegraded":"2023-11-16T23:58:51.975877+0000","last_fullsized":"2023-11-16T23:58:51.975877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:48:34.246739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973742+0000","last_change":"2023-11-16T23:58:50.927654+0000","last_active":"2023-11-16T23:58:51.973742+0000","last_peered":"2023-11-16T23:58:51.973742+0000","last_clean":"2023-11-16T23:58:51.973742+0000","last_became_active":"2023-11-16T23:58:50.926936+0000","last_became_peered":"2023-11-16T23:58:50.926936+0000","last_unstale":"2023-11-16T23:58:51.973742+0000","last_undegraded":"2023-11-16T23:58:51.973742+0000","last_fullsized":"2023-11-16T23:58:51.973742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:20.080380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:41.067051+0000","last_change":"2023-11-16T23:58:45.717981+0000","last_active":"2023-11-16T23:59:41.067051+0000","last_peered":"2023-11-16T23:59:41.067051+0000","last_clean":"2023-11-16T23:59:41.067051+0000","last_became_active":"2023-11-16T23:58:45.717702+0000","last_became_peered":"2023-11-16T23:58:45.717702+0000","last_unstale":"2023-11-16T23:59:41.067051+0000","last_undegraded":"2023-11-16T23:59:41.067051+0000","last_fullsized":"2023-11-16T23:59:41.067051+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:32:06.566478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974772+0000","last_change":"2023-11-16T23:58:46.288768+0000","last_active":"2023-11-16T23:58:51.974772+0000","last_peered":"2023-11-16T23:58:51.974772+0000","last_clean":"2023-11-16T23:58:51.974772+0000","last_became_active":"2023-11-16T23:58:46.288430+0000","last_became_peered":"2023-11-16T23:58:46.288430+0000","last_unstale":"2023-11-16T23:58:51.974772+0000","last_undegraded":"2023-11-16T23:58:51.974772+0000","last_fullsized":"2023-11-16T23:58:51.974772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:19:06.093201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416385+0000","last_change":"2023-11-16T23:58:50.724748+0000","last_active":"2023-11-16T23:58:51.416385+0000","last_peered":"2023-11-16T23:58:51.416385+0000","last_clean":"2023-11-16T23:58:51.416385+0000","last_became_active":"2023-11-16T23:58:50.724313+0000","last_became_peered":"2023-11-16T23:58:50.724313+0000","last_unstale":"2023-11-16T23:58:51.416385+0000","last_undegraded":"2023-11-16T23:58:51.416385+0000","last_fullsized":"2023-11-16T23:58:51.416385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:17:21.820364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732623+0000","last_change":"2023-11-16T23:58:45.715563+0000","last_active":"2023-11-16T23:59:32.732623+0000","last_peered":"2023-11-16T23:59:32.732623+0000","last_clean":"2023-11-16T23:59:32.732623+0000","last_became_active":"2023-11-16T23:58:45.715217+0000","last_became_peered":"2023-11-16T23:58:45.715217+0000","last_unstale":"2023-11-16T23:59:32.732623+0000","last_undegraded":"2023-11-16T23:59:32.732623+0000","last_fullsized":"2023-11-16T23:59:32.732623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:41.330842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242993+0000","last_change":"2023-11-16T23:58:46.298305+0000","last_active":"2023-11-16T23:58:51.242993+0000","last_peered":"2023-11-16T23:58:51.242993+0000","last_clean":"2023-11-16T23:58:51.242993+0000","last_became_active":"2023-11-16T23:58:46.297444+0000","last_became_peered":"2023-11-16T23:58:46.297444+0000","last_unstale":"2023-11-16T23:58:51.242993+0000","last_undegraded":"2023-11-16T23:58:51.242993+0000","last_fullsized":"2023-11-16T23:58:51.242993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:36.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974077+0000","last_change":"2023-11-16T23:58:50.927915+0000","last_active":"2023-11-16T23:58:51.974077+0000","last_peered":"2023-11-16T23:58:51.974077+0000","last_clean":"2023-11-16T23:58:51.974077+0000","last_became_active":"2023-11-16T23:58:50.927451+0000","last_became_peered":"2023-11-16T23:58:50.927451+0000","last_unstale":"2023-11-16T23:58:51.974077+0000","last_undegraded":"2023-11-16T23:58:51.974077+0000","last_fullsized":"2023-11-16T23:58:51.974077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:21:21.865022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732946+0000","last_change":"2023-11-16T23:58:45.710870+0000","last_active":"2023-11-16T23:59:32.732946+0000","last_peered":"2023-11-16T23:59:32.732946+0000","last_clean":"2023-11-16T23:59:32.732946+0000","last_became_active":"2023-11-16T23:58:45.710536+0000","last_became_peered":"2023-11-16T23:58:45.710536+0000","last_unstale":"2023-11-16T23:59:32.732946+0000","last_undegraded":"2023-11-16T23:59:32.732946+0000","last_fullsized":"2023-11-16T23:59:32.732946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:32.891477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973290+0000","last_change":"2023-11-16T23:58:46.282277+0000","last_active":"2023-11-16T23:58:51.973290+0000","last_peered":"2023-11-16T23:58:51.973290+0000","last_clean":"2023-11-16T23:58:51.973290+0000","last_became_active":"2023-11-16T23:58:46.281913+0000","last_became_peered":"2023-11-16T23:58:46.281913+0000","last_unstale":"2023-11-16T23:58:51.973290+0000","last_undegraded":"2023-11-16T23:58:51.973290+0000","last_fullsized":"2023-11-16T23:58:51.973290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:19:12.760160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976155+0000","last_change":"2023-11-16T23:58:50.740914+0000","last_active":"2023-11-16T23:58:51.976155+0000","last_peered":"2023-11-16T23:58:51.976155+0000","last_clean":"2023-11-16T23:58:51.976155+0000","last_became_active":"2023-11-16T23:58:50.740348+0000","last_became_peered":"2023-11-16T23:58:50.740348+0000","last_unstale":"2023-11-16T23:58:51.976155+0000","last_undegraded":"2023-11-16T23:58:51.976155+0000","last_fullsized":"2023-11-16T23:58:51.976155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:59:30.440820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734107+0000","last_change":"2023-11-16T23:58:45.723953+0000","last_active":"2023-11-16T23:59:32.734107+0000","last_peered":"2023-11-16T23:59:32.734107+0000","last_clean":"2023-11-16T23:59:32.734107+0000","last_became_active":"2023-11-16T23:58:45.722075+0000","last_became_peered":"2023-11-16T23:58:45.722075+0000","last_unstale":"2023-11-16T23:59:32.734107+0000","last_undegraded":"2023-11-16T23:59:32.734107+0000","last_fullsized":"2023-11-16T23:59:32.734107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:46:29.950815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240548+0000","last_change":"2023-11-16T23:58:46.290530+0000","last_active":"2023-11-16T23:58:51.240548+0000","last_peered":"2023-11-16T23:58:51.240548+0000","last_clean":"2023-11-16T23:58:51.240548+0000","last_became_active":"2023-11-16T23:58:46.290181+0000","last_became_peered":"2023-11-16T23:58:46.290181+0000","last_unstale":"2023-11-16T23:58:51.240548+0000","last_undegraded":"2023-11-16T23:58:51.240548+0000","last_fullsized":"2023-11-16T23:58:51.240548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:20.504586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974154+0000","last_change":"2023-11-16T23:58:50.739559+0000","last_active":"2023-11-16T23:58:51.974154+0000","last_peered":"2023-11-16T23:58:51.974154+0000","last_clean":"2023-11-16T23:58:51.974154+0000","last_became_active":"2023-11-16T23:58:50.739156+0000","last_became_peered":"2023-11-16T23:58:50.739156+0000","last_unstale":"2023-11-16T23:58:51.974154+0000","last_undegraded":"2023-11-16T23:58:51.974154+0000","last_fullsized":"2023-11-16T23:58:51.974154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:45.312325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416494+0000","last_change":"2023-11-16T23:58:45.723679+0000","last_active":"2023-11-16T23:58:51.416494+0000","last_peered":"2023-11-16T23:58:51.416494+0000","last_clean":"2023-11-16T23:58:51.416494+0000","last_became_active":"2023-11-16T23:58:45.723340+0000","last_became_peered":"2023-11-16T23:58:45.723340+0000","last_unstale":"2023-11-16T23:58:51.416494+0000","last_undegraded":"2023-11-16T23:58:51.416494+0000","last_fullsized":"2023-11-16T23:58:51.416494+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:28:17.612030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974645+0000","last_change":"2023-11-16T23:58:46.297004+0000","last_active":"2023-11-16T23:58:51.974645+0000","last_peered":"2023-11-16T23:58:51.974645+0000","last_clean":"2023-11-16T23:58:51.974645+0000","last_became_active":"2023-11-16T23:58:46.296622+0000","last_became_peered":"2023-11-16T23:58:46.296622+0000","last_unstale":"2023-11-16T23:58:51.974645+0000","last_undegraded":"2023-11-16T23:58:51.974645+0000","last_fullsized":"2023-11-16T23:58:51.974645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:00:00.033560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974484+0000","last_change":"2023-11-16T23:58:50.927313+0000","last_active":"2023-11-16T23:58:51.974484+0000","last_peered":"2023-11-16T23:58:51.974484+0000","last_clean":"2023-11-16T23:58:51.974484+0000","last_became_active":"2023-11-16T23:58:50.926865+0000","last_became_peered":"2023-11-16T23:58:50.926865+0000","last_unstale":"2023-11-16T23:58:51.974484+0000","last_undegraded":"2023-11-16T23:58:51.974484+0000","last_fullsized":"2023-11-16T23:58:51.974484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:41:57.718515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733211+0000","last_change":"2023-11-16T23:58:45.714188+0000","last_active":"2023-11-16T23:59:32.733211+0000","last_peered":"2023-11-16T23:59:32.733211+0000","last_clean":"2023-11-16T23:59:32.733211+0000","last_became_active":"2023-11-16T23:58:45.713785+0000","last_became_peered":"2023-11-16T23:58:45.713785+0000","last_unstale":"2023-11-16T23:59:32.733211+0000","last_undegraded":"2023-11-16T23:59:32.733211+0000","last_fullsized":"2023-11-16T23:59:32.733211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:49:36.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239324+0000","last_change":"2023-11-16T23:58:46.291307+0000","last_active":"2023-11-16T23:58:51.239324+0000","last_peered":"2023-11-16T23:58:51.239324+0000","last_clean":"2023-11-16T23:58:51.239324+0000","last_became_active":"2023-11-16T23:58:46.289946+0000","last_became_peered":"2023-11-16T23:58:46.289946+0000","last_unstale":"2023-11-16T23:58:51.239324+0000","last_undegraded":"2023-11-16T23:58:51.239324+0000","last_fullsized":"2023-11-16T23:58:51.239324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:58:49.392838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973947+0000","last_change":"2023-11-16T23:58:50.737205+0000","last_active":"2023-11-16T23:58:51.973947+0000","last_peered":"2023-11-16T23:58:51.973947+0000","last_clean":"2023-11-16T23:58:51.973947+0000","last_became_active":"2023-11-16T23:58:50.736756+0000","last_became_peered":"2023-11-16T23:58:50.736756+0000","last_unstale":"2023-11-16T23:58:51.973947+0000","last_undegraded":"2023-11-16T23:58:51.973947+0000","last_fullsized":"2023-11-16T23:58:51.973947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:05:08.988766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.719115+0000","last_change":"2023-11-16T23:58:45.713979+0000","last_active":"2023-11-16T23:59:33.719115+0000","last_peered":"2023-11-16T23:59:33.719115+0000","last_clean":"2023-11-16T23:59:33.719115+0000","last_became_active":"2023-11-16T23:58:45.713492+0000","last_became_peered":"2023-11-16T23:58:45.713492+0000","last_unstale":"2023-11-16T23:59:33.719115+0000","last_undegraded":"2023-11-16T23:59:33.719115+0000","last_fullsized":"2023-11-16T23:59:33.719115+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:17:15.481986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239735+0000","last_change":"2023-11-16T23:58:46.289100+0000","last_active":"2023-11-16T23:58:51.239735+0000","last_peered":"2023-11-16T23:58:51.239735+0000","last_clean":"2023-11-16T23:58:51.239735+0000","last_became_active":"2023-11-16T23:58:46.288506+0000","last_became_peered":"2023-11-16T23:58:46.288506+0000","last_unstale":"2023-11-16T23:58:51.239735+0000","last_undegraded":"2023-11-16T23:58:51.239735+0000","last_fullsized":"2023-11-16T23:58:51.239735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:44:12.486352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242604+0000","last_change":"2023-11-16T23:58:50.741205+0000","last_active":"2023-11-16T23:58:51.242604+0000","last_peered":"2023-11-16T23:58:51.242604+0000","last_clean":"2023-11-16T23:58:51.242604+0000","last_became_active":"2023-11-16T23:58:50.740871+0000","last_became_peered":"2023-11-16T23:58:50.740871+0000","last_unstale":"2023-11-16T23:58:51.242604+0000","last_undegraded":"2023-11-16T23:58:51.242604+0000","last_fullsized":"2023-11-16T23:58:51.242604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:25:17.203133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584292+0000","last_change":"2023-11-16T23:58:45.723618+0000","last_active":"2023-11-16T23:58:51.584292+0000","last_peered":"2023-11-16T23:58:51.584292+0000","last_clean":"2023-11-16T23:58:51.584292+0000","last_became_active":"2023-11-16T23:58:45.723266+0000","last_became_peered":"2023-11-16T23:58:45.723266+0000","last_unstale":"2023-11-16T23:58:51.584292+0000","last_undegraded":"2023-11-16T23:58:51.584292+0000","last_fullsized":"2023-11-16T23:58:51.584292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:20:44.206104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239349+0000","last_change":"2023-11-16T23:58:46.295295+0000","last_active":"2023-11-16T23:58:51.239349+0000","last_peered":"2023-11-16T23:58:51.239349+0000","last_clean":"2023-11-16T23:58:51.239349+0000","last_became_active":"2023-11-16T23:58:46.294801+0000","last_became_peered":"2023-11-16T23:58:46.294801+0000","last_unstale":"2023-11-16T23:58:51.239349+0000","last_undegraded":"2023-11-16T23:58:51.239349+0000","last_fullsized":"2023-11-16T23:58:51.239349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:00.523212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583352+0000","last_change":"2023-11-16T23:58:50.725856+0000","last_active":"2023-11-16T23:58:51.583352+0000","last_peered":"2023-11-16T23:58:51.583352+0000","last_clean":"2023-11-16T23:58:51.583352+0000","last_became_active":"2023-11-16T23:58:50.725453+0000","last_became_peered":"2023-11-16T23:58:50.725453+0000","last_unstale":"2023-11-16T23:58:51.583352+0000","last_undegraded":"2023-11-16T23:58:51.583352+0000","last_fullsized":"2023-11-16T23:58:51.583352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:32:17.547808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976506+0000","last_change":"2023-11-16T23:58:45.718226+0000","last_active":"2023-11-16T23:58:51.976506+0000","last_peered":"2023-11-16T23:58:51.976506+0000","last_clean":"2023-11-16T23:58:51.976506+0000","last_became_active":"2023-11-16T23:58:45.717893+0000","last_became_peered":"2023-11-16T23:58:45.717893+0000","last_unstale":"2023-11-16T23:58:51.976506+0000","last_undegraded":"2023-11-16T23:58:51.976506+0000","last_fullsized":"2023-11-16T23:58:51.976506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:01.114364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239740+0000","last_change":"2023-11-16T23:58:46.289906+0000","last_active":"2023-11-16T23:58:51.239740+0000","last_peered":"2023-11-16T23:58:51.239740+0000","last_clean":"2023-11-16T23:58:51.239740+0000","last_became_active":"2023-11-16T23:58:46.289266+0000","last_became_peered":"2023-11-16T23:58:46.289266+0000","last_unstale":"2023-11-16T23:58:51.239740+0000","last_undegraded":"2023-11-16T23:58:51.239740+0000","last_fullsized":"2023-11-16T23:58:51.239740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:54:03.586113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240620+0000","last_change":"2023-11-16T23:58:46.292220+0000","last_active":"2023-11-16T23:58:51.240620+0000","last_peered":"2023-11-16T23:58:51.240620+0000","last_clean":"2023-11-16T23:58:51.240620+0000","last_became_active":"2023-11-16T23:58:46.291840+0000","last_became_peered":"2023-11-16T23:58:46.291840+0000","last_unstale":"2023-11-16T23:58:51.240620+0000","last_undegraded":"2023-11-16T23:58:51.240620+0000","last_fullsized":"2023-11-16T23:58:51.240620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:18:35.280653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416702+0000","last_change":"2023-11-16T23:58:46.287601+0000","last_active":"2023-11-16T23:58:51.416702+0000","last_peered":"2023-11-16T23:58:51.416702+0000","last_clean":"2023-11-16T23:58:51.416702+0000","last_became_active":"2023-11-16T23:58:46.284738+0000","last_became_peered":"2023-11-16T23:58:46.284738+0000","last_unstale":"2023-11-16T23:58:51.416702+0000","last_undegraded":"2023-11-16T23:58:51.416702+0000","last_fullsized":"2023-11-16T23:58:51.416702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:49.913458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416384+0000","last_change":"2023-11-16T23:58:46.288601+0000","last_active":"2023-11-16T23:58:51.416384+0000","last_peered":"2023-11-16T23:58:51.416384+0000","last_clean":"2023-11-16T23:58:51.416384+0000","last_became_active":"2023-11-16T23:58:46.288266+0000","last_became_peered":"2023-11-16T23:58:46.288266+0000","last_unstale":"2023-11-16T23:58:51.416384+0000","last_undegraded":"2023-11-16T23:58:51.416384+0000","last_fullsized":"2023-11-16T23:58:51.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:18.659483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242778+0000","last_change":"2023-11-16T23:58:46.286495+0000","last_active":"2023-11-16T23:58:51.242778+0000","last_peered":"2023-11-16T23:58:51.242778+0000","last_clean":"2023-11-16T23:58:51.242778+0000","last_became_active":"2023-11-16T23:58:46.281157+0000","last_became_peered":"2023-11-16T23:58:46.281157+0000","last_unstale":"2023-11-16T23:58:51.242778+0000","last_undegraded":"2023-11-16T23:58:51.242778+0000","last_fullsized":"2023-11-16T23:58:51.242778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:02.565515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976001+0000","last_change":"2023-11-16T23:58:46.282373+0000","last_active":"2023-11-16T23:58:51.976001+0000","last_peered":"2023-11-16T23:58:51.976001+0000","last_clean":"2023-11-16T23:58:51.976001+0000","last_became_active":"2023-11-16T23:58:46.282028+0000","last_became_peered":"2023-11-16T23:58:46.282028+0000","last_unstale":"2023-11-16T23:58:51.976001+0000","last_undegraded":"2023-11-16T23:58:51.976001+0000","last_fullsized":"2023-11-16T23:58:51.976001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:29:48.640544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241073+0000","last_change":"2023-11-16T23:58:46.285528+0000","last_active":"2023-11-16T23:58:51.241073+0000","last_peered":"2023-11-16T23:58:51.241073+0000","last_clean":"2023-11-16T23:58:51.241073+0000","last_became_active":"2023-11-16T23:58:46.285227+0000","last_became_peered":"2023-11-16T23:58:46.285227+0000","last_unstale":"2023-11-16T23:58:51.241073+0000","last_undegraded":"2023-11-16T23:58:51.241073+0000","last_fullsized":"2023-11-16T23:58:51.241073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:45:11.259431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416982+0000","last_change":"2023-11-16T23:58:46.283514+0000","last_active":"2023-11-16T23:58:51.416982+0000","last_peered":"2023-11-16T23:58:51.416982+0000","last_clean":"2023-11-16T23:58:51.416982+0000","last_became_active":"2023-11-16T23:58:46.283026+0000","last_became_peered":"2023-11-16T23:58:46.283026+0000","last_unstale":"2023-11-16T23:58:51.416982+0000","last_undegraded":"2023-11-16T23:58:51.416982+0000","last_fullsized":"2023-11-16T23:58:51.416982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:14.255094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974891+0000","last_change":"2023-11-16T23:58:46.283334+0000","last_active":"2023-11-16T23:58:51.974891+0000","last_peered":"2023-11-16T23:58:51.974891+0000","last_clean":"2023-11-16T23:58:51.974891+0000","last_became_active":"2023-11-16T23:58:46.283012+0000","last_became_peered":"2023-11-16T23:58:46.283012+0000","last_unstale":"2023-11-16T23:58:51.974891+0000","last_undegraded":"2023-11-16T23:58:51.974891+0000","last_fullsized":"2023-11-16T23:58:51.974891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:44.161087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583457+0000","last_change":"2023-11-16T23:58:46.292720+0000","last_active":"2023-11-16T23:58:51.583457+0000","last_peered":"2023-11-16T23:58:51.583457+0000","last_clean":"2023-11-16T23:58:51.583457+0000","last_became_active":"2023-11-16T23:58:46.292109+0000","last_became_peered":"2023-11-16T23:58:46.292109+0000","last_unstale":"2023-11-16T23:58:51.583457+0000","last_undegraded":"2023-11-16T23:58:51.583457+0000","last_fullsized":"2023-11-16T23:58:51.583457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:41:12.067371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974297+0000","last_change":"2023-11-16T23:58:46.279919+0000","last_active":"2023-11-16T23:58:51.974297+0000","last_peered":"2023-11-16T23:58:51.974297+0000","last_clean":"2023-11-16T23:58:51.974297+0000","last_became_active":"2023-11-16T23:58:46.279601+0000","last_became_peered":"2023-11-16T23:58:46.279601+0000","last_unstale":"2023-11-16T23:58:51.974297+0000","last_undegraded":"2023-11-16T23:58:51.974297+0000","last_fullsized":"2023-11-16T23:58:51.974297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:28:22.080722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975063+0000","last_change":"2023-11-16T23:58:46.293091+0000","last_active":"2023-11-16T23:58:51.975063+0000","last_peered":"2023-11-16T23:58:51.975063+0000","last_clean":"2023-11-16T23:58:51.975063+0000","last_became_active":"2023-11-16T23:58:46.292728+0000","last_became_peered":"2023-11-16T23:58:46.292728+0000","last_unstale":"2023-11-16T23:58:51.975063+0000","last_undegraded":"2023-11-16T23:58:51.975063+0000","last_fullsized":"2023-11-16T23:58:51.975063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:57:24.095879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975716+0000","last_change":"2023-11-16T23:58:46.281708+0000","last_active":"2023-11-16T23:58:51.975716+0000","last_peered":"2023-11-16T23:58:51.975716+0000","last_clean":"2023-11-16T23:58:51.975716+0000","last_became_active":"2023-11-16T23:58:46.281335+0000","last_became_peered":"2023-11-16T23:58:46.281335+0000","last_unstale":"2023-11-16T23:58:51.975716+0000","last_undegraded":"2023-11-16T23:58:51.975716+0000","last_fullsized":"2023-11-16T23:58:51.975716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:46:24.208579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240805+0000","last_change":"2023-11-16T23:58:46.286046+0000","last_active":"2023-11-16T23:58:51.240805+0000","last_peered":"2023-11-16T23:58:51.240805+0000","last_clean":"2023-11-16T23:58:51.240805+0000","last_became_active":"2023-11-16T23:58:46.285702+0000","last_became_peered":"2023-11-16T23:58:46.285702+0000","last_unstale":"2023-11-16T23:58:51.240805+0000","last_undegraded":"2023-11-16T23:58:51.240805+0000","last_fullsized":"2023-11-16T23:58:51.240805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:15:25.406768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64297,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":297,"num_read_kb":218,"num_write":157,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":192891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":79,"seq":339302416414,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":974848,"data_stored":475644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86199999999999999,"5min":0.874,"15min":0.874},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.49399999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.56000000000000005,"5min":0.433,"15min":0.433},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85399999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.629}]},{"osd":5,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79700000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.56100000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.753},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72999999999999998}]},{"osd":8,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64500000000000002}]},{"osd":9,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76900000000000002,"15min":0.76600000000000001},"min":{"1min":0.65000000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.94899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77400000000000002,"15min":0.76900000000000002},"min":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.96099999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.78000000000000003}]},{"osd":10,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.86899999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.752,"15min":0.751},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.90300000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78600000000000003}]}]},{"osd":10,"up_from":71,"seq":304942678051,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7068,"kb_used_data":1620,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738084,"statfs":{"total":95995035648,"available":95987798016,"internally_reserved":0,"allocated":1658880,"data_stored":1122926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.47299999999999998,"15min":0.47699999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79700000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79600000000000004,"5min":0.877,"15min":0.877},"last":0.746},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.36299999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.754,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.70299999999999996},"min":{"1min":0.54200000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.85399999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.622}]},{"osd":6,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83799999999999997,"5min":1.103,"15min":1.103},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91300000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.434},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71599999999999997,"15min":0.71999999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.80100000000000005,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94099999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.77900000000000003}]},{"osd":9,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.97799999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.84799999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.81699999999999995}]},{"osd":11,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90500000000000003,"5min":1.034,"15min":1.034},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79600000000000004,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.89000000000000001,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.71799999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841579,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6292,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738860,"statfs":{"total":95995035648,"available":95988592640,"internally_reserved":0,"allocated":995328,"data_stored":479976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.728,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.58599999999999997,"15min":0.61399999999999999},"min":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":1.5469999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.8320000000000001,"5min":1.899,"15min":1.899},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.78200000000000003,"15min":0.80700000000000005},"min":{"1min":0.437,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.76500000000000001,"15min":0.78100000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69099999999999995}]},{"osd":3,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.65900000000000003},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.7030000000000001,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.437},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.80100000000000005,"15min":0.84299999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.794,"5min":2.056,"15min":2.056},"last":0.65700000000000003}]},{"osd":4,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.81299999999999994,"15min":0.85899999999999999},"min":{"1min":0.48799999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.499},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.82199999999999995},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6299999999999999,"5min":2.677,"15min":2.677},"last":0.67200000000000004}]},{"osd":5,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.872,"15min":0.91900000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2310000000000001,"5min":1.329,"15min":1.329},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.875,"15min":0.88300000000000001},"min":{"1min":0.63,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5860000000000001,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.76500000000000001}]},{"osd":6,"last update":"Thu Nov 16 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.86299999999999999}]},{"osd":7,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.97099999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.6699999999999999,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.98099999999999998,"15min":1.0249999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9399999999999999,"5min":1.98,"15min":1.98},"last":0.58199999999999996}]},{"osd":8,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.98099999999999998,"15min":1.0269999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.008,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.84899999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.874,"15min":0.878},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0649999999999999,"5min":1.141,"15min":1.141},"last":0.54000000000000004}]},{"osd":11,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.753},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.80900000000000005}]}]},{"osd":8,"up_from":58,"seq":249108103216,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6480,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738672,"statfs":{"total":95995035648,"available":95988400128,"internally_reserved":0,"allocated":991232,"data_stored":475800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.879,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57099999999999995,"15min":0.58099999999999996},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59999999999999998,"15min":0.59899999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89600000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.49299999999999999},"min":{"1min":0.312,"5min":0.222,"15min":0.222},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.497},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68200000000000005,"15min":0.66800000000000004},"min":{"1min":0.42599999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.86199999999999999,"5min":4.5149999999999997,"15min":4.5149999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84499999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.626}]},{"osd":5,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54800000000000004,"15min":0.56399999999999995},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83499999999999996,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60299999999999998,"15min":0.63500000000000001},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80700000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.72799999999999998},"min":{"1min":0.45100000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.749},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76100000000000001,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92000000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.89800000000000002}]},{"osd":7,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68600000000000005,"15min":0.69799999999999995},"min":{"1min":0.47499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.081,"15min":1.081},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.79500000000000004}]},{"osd":9,"last update":"Fri Nov 17 00:00:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.77000000000000002,"15min":0.77100000000000002},"min":{"1min":0.63100000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.04,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.78800000000000003,"15min":0.77200000000000002},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.997,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":0.875}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98399999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.141,"5min":1.145,"15min":1.145},"last":0.82799999999999996}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.81100000000000005}]}]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7124,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738028,"statfs":{"total":95995035648,"available":95987740672,"internally_reserved":0,"allocated":1585152,"data_stored":1071612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.53600000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81599999999999995,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57299999999999995,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77800000000000002,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.624},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.94599999999999995,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.98899999999999999}]},{"osd":4,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.501,"15min":0.501},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.503,"15min":0.505},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.44400000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.45900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.71599999999999997}]},{"osd":6,"last update":"Thu Nov 16 23:59:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96199999999999997,"5min":1.069,"15min":1.069},"last":0.67400000000000004}]},{"osd":8,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64100000000000001,"15min":0.61399999999999999},"min":{"1min":0.499,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90300000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.64600000000000002,"15min":0.61799999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86299999999999999,"5min":1.071,"15min":1.071},"last":0.78700000000000003}]},{"osd":9,"last update":"Fri Nov 17 00:00:04 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92300000000000004,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.61099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0109999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.80100000000000005}]},{"osd":10,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.51700000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.77200000000000002}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73499999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6396,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738756,"statfs":{"total":95995035648,"available":95988486144,"internally_reserved":0,"allocated":970752,"data_stored":471056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.52500000000000002,"15min":0.501},"min":{"1min":0.38400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.375,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.66500000000000004,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.60999999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.76200000000000001}]},{"osd":2,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67800000000000005,"15min":0.65100000000000002},"min":{"1min":0.52500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.57099999999999995,"5min":0.432,"15min":0.432},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":0.621},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.52200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.746,"15min":0.76000000000000001},"min":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.80400000000000005,"15min":0.79700000000000004},"min":{"1min":0.67500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.78600000000000003,"15min":0.76600000000000001},"min":{"1min":0.56100000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Nov 17 00:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.624,"5min":0.443,"15min":0.443},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.81499999999999995,"15min":0.78300000000000003},"min":{"1min":0.66000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.878}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.86499999999999999,"15min":0.86799999999999999},"min":{"1min":0.626,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.86499999999999999,"15min":0.81899999999999995},"min":{"1min":0.65100000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.256,"5min":5.5839999999999996,"15min":5.5839999999999996},"last":0.93000000000000005}]},{"osd":9,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.94299999999999995,"15min":0.93500000000000005},"min":{"1min":0.58399999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.98699999999999999}]},{"osd":10,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93700000000000006,"15min":0.93500000000000005},"min":{"1min":0.66900000000000004,"5min":0.63,"15min":0.63},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.90500000000000003,"15min":0.90100000000000002},"min":{"1min":0.70599999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.3180000000000001,"5min":1.351,"15min":1.351},"last":0.96599999999999997}]},{"osd":11,"last update":"Thu Nov 16 23:59:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":1.0389999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":1036288,"data_stored":531359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.51600000000000001},"min":{"1min":0.41099999999999998,"5min":0.314,"15min":0.30099999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":0.998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59899999999999998,"15min":0.51900000000000002},"min":{"1min":0.46400000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84299999999999997,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.54000000000000004}]},{"osd":2,"last update":"Fri Nov 17 00:00:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59499999999999997,"15min":0.55500000000000005},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":0.878,"15min":0.878},"last":0.749},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.501,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83299999999999996,"5min":1.054,"15min":1.054},"last":0.65800000000000003}]},{"osd":3,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.64400000000000002}]},{"osd":4,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.71199999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.40799999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.628,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.55600000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.877,"5min":1.103,"15min":1.103},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96499999999999997,"5min":1.159,"15min":1.159},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.89500000000000002,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57999999999999996,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.73599999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68700000000000006,"15min":0.64400000000000002},"min":{"1min":0.54100000000000004,"5min":0.37,"15min":0.37},"max":{"1min":0.94199999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68999999999999995,"15min":0.65000000000000002},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94299999999999995,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.82499999999999996}]},{"osd":9,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.627,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93100000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0049999999999999,"5min":1.173,"15min":1.173},"last":0.86599999999999999}]},{"osd":10,"last update":"Thu Nov 16 23:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78700000000000003,"15min":0.79100000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0149999999999999,"5min":1.133,"15min":1.133},"last":0.72599999999999998}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.88100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":472444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61199999999999999,"15min":0.55900000000000005},"min":{"1min":0.46100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":1.454},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.64400000000000002,"15min":0.57999999999999996},"min":{"1min":0.48099999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.89300000000000002}]},{"osd":2,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.625,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.29099999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.61599999999999999},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.35699999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":1.5509999999999999}]},{"osd":3,"last update":"Thu Nov 16 23:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":1.23},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.94799999999999995}]},{"osd":4,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.71699999999999997,"15min":0.63700000000000001},"min":{"1min":0.61399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.032},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.436,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Nov 16 23:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.61199999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":1.073},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67200000000000004,"15min":0.65500000000000003},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.79300000000000004,"15min":0.72799999999999998},"min":{"1min":0.59199999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":1.579}]},{"osd":7,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":1.5069999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.874,"15min":0.872},"min":{"1min":0.55700000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.113},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.175,"5min":1.276,"15min":1.276},"last":0.624}]},{"osd":9,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.879},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.6759999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93500000000000005,"15min":0.93300000000000005},"min":{"1min":0.67800000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.194,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.6399999999999999}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.90200000000000002,"15min":0.874},"min":{"1min":0.68300000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":1.153},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.85299999999999998,"15min":0.83899999999999997},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":1.609}]},{"osd":11,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.872,"15min":0.85099999999999998},"min":{"1min":0.68899999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":1.7110000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.94699999999999995,"15min":0.93899999999999995},"min":{"1min":0.79200000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3160000000000001,"5min":1.385,"15min":1.385},"last":1.1899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7572,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737580,"statfs":{"total":95995035648,"available":95987281920,"internally_reserved":0,"allocated":1060864,"data_stored":533051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 00:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.749},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78200000000000003,"5min":1.675,"15min":1.675},"last":0.60099999999999998}]},{"osd":1,"last update":"Fri Nov 17 00:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.501},{"interface":"front","average":{"1min":0.622,"5min":0.55100000000000005,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.433}]},{"osd":3,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Nov 16 23:59:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Nov 16 23:59:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.57299999999999995,"5min":0.37,"15min":0.37},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.624,"15min":0.60199999999999998},"min":{"1min":0.56200000000000006,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Nov 16 23:59:49 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.88700000000000001}]},{"osd":8,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.64800000000000002,"15min":0.63800000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.73699999999999999}]},{"osd":9,"last update":"Fri Nov 17 00:00:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73499999999999999,"15min":0.71699999999999997},"min":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96599999999999997,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.74199999999999999,"15min":0.71799999999999997},"min":{"1min":0.65000000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.93600000000000005,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.83199999999999996}]},{"osd":10,"last update":"Thu Nov 16 23:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.79800000000000004}]},{"osd":11,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.873},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.84599999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":470919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.44900000000000001,"15min":0.40699999999999997},"min":{"1min":0.34399999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.51900000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65500000000000003,"15min":0.59899999999999998},"min":{"1min":0.55100000000000005,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94799999999999995,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71399999999999997,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.88500000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Nov 17 00:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.72099999999999997,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.68899999999999995,"15min":0.629},"min":{"1min":0.52300000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.88}]},{"osd":5,"last update":"Thu Nov 16 23:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69599999999999995,"15min":0.63900000000000001},"min":{"1min":0.51400000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.1220000000000001,"5min":1.492,"15min":1.492},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68999999999999995,"15min":0.64600000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.212,"15min":1.212},"last":0.85699999999999998}]},{"osd":6,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60599999999999998,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.016,"5min":1.093,"15min":1.093},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.69599999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.214,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.55200000000000005}]},{"osd":7,"last update":"Thu Nov 16 23:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.79800000000000004},"min":{"1min":0.66300000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1739999999999999,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.75},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77600000000000002,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1419999999999999,"5min":1.256,"15min":1.256},"last":0.91100000000000003}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.90600000000000003,"15min":0.86799999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":7.173,"5min":7.173,"15min":7.173},"last":0.84499999999999997}]},{"osd":9,"last update":"Fri Nov 17 00:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.97199999999999998,"15min":1.0049999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.377,"5min":1.617,"15min":1.617},"last":0.77200000000000002}]},{"osd":10,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.212,"15min":1.2609999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.212,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.89500000000000002,"15min":0.90300000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.1919999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.93100000000000005}]},{"osd":11,"last update":"Thu Nov 16 23:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.78700000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920834,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":983040,"data_stored":475658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40500000000000003,"15min":0.36699999999999999},"min":{"1min":0.35599999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48599999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.40200000000000002,"15min":0.372},"min":{"1min":0.34999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.45100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57199999999999995,"15min":0.54200000000000004},"min":{"1min":0.498,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67300000000000004,"5min":0.872,"15min":0.872},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64800000000000002,"15min":0.59299999999999997},"min":{"1min":0.54400000000000004,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76700000000000002,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65000000000000002,"15min":0.621},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78700000000000003,"5min":1.105,"15min":1.105},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Nov 16 23:59:31 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Nov 17 00:00:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.60299999999999998},"min":{"1min":0.495,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93400000000000005,"5min":0.997,"15min":0.997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.73899999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83899999999999997}]},{"osd":9,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.72699999999999998},"min":{"1min":0.52800000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.97999999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.995,"5min":1.05,"15min":1.05},"last":0.90100000000000002}]},{"osd":10,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.51100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88900000000000001,"5min":1.083,"15min":1.083},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78700000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.75800000000000001}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.82799999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":1589248,"data_stored":1066917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52000000000000002},"min":{"1min":0.42199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.753},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.78000000000000003}]},{"osd":1,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.753,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40400000000000003,"15min":0.38800000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55700000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.38700000000000001},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.47099999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.621}]},{"osd":4,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.55300000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81299999999999994,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.44,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Nov 16 23:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63,"15min":0.62},"min":{"1min":0.443,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.76600000000000001}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63700000000000001,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.44500000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64500000000000002}]},{"osd":8,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.45500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.86199999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53500000000000003}]},{"osd":9,"last update":"Fri Nov 17 00:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.57699999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71799999999999997,"15min":0.70899999999999996},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.91200000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68999999999999995}]},{"osd":10,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.66900000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.72399999999999998,"15min":0.71699999999999997},"min":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T00:00:23.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T00:00:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:24.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:24 smithi023 ceph-mon[97824]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:24.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:24 smithi023 ceph-mon[97824]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:24.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:24 smithi173 ceph-mon[131534]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:24 smithi173 ceph-mon[131534]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:24 smithi156 ceph-mon[131105]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:24 smithi156 ceph-mon[131105]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.402+0000 7fbd888f6700 1 -- 172.21.15.23:0/1959493704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd841011a0 msgr2=0x7fbd84101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.402+0000 7fbd888f6700 1 --2- 172.21.15.23:0/1959493704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd841011a0 0x7fbd84101570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009990 tx=0x7fbd74009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 -- 172.21.15.23:0/1959493704 shutdown_connections 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 --2- 172.21.15.23:0/1959493704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd8410a6c0 0x7fbd8410cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 --2- 172.21.15.23:0/1959493704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd841011a0 0x7fbd84101570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 --2- 172.21.15.23:0/1959493704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd84101ab0 0x7fbd8410a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 -- 172.21.15.23:0/1959493704 >> 172.21.15.23:0/1959493704 conn(0x7fbd840761a0 msgr2=0x7fbd840765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 -- 172.21.15.23:0/1959493704 shutdown_connections 2023-11-17T00:00:25.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.403+0000 7fbd888f6700 1 -- 172.21.15.23:0/1959493704 wait complete. 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 Processor -- start 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 -- start start 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 0x7fbd841962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd8410a6c0 0x7fbd8419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8419b270 con 0x7fbd84101ab0 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8419b3e0 con 0x7fbd841011a0 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd888f6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8419b5a0 con 0x7fbd8410a6c0 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.404+0000 7fbd8259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44446/0 (socket says 172.21.15.23:44446) 2023-11-17T00:00:25.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 -- 172.21.15.23:0/3805410740 learned_addr learned my addr 172.21.15.23:0/3805410740 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd82d9d700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 0x7fbd841962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8359e700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd8410a6c0 0x7fbd8419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 -- 172.21.15.23:0/3805410740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd8410a6c0 msgr2=0x7fbd8419ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd8410a6c0 0x7fbd8419ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 -- 172.21.15.23:0/3805410740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 msgr2=0x7fbd841962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 0x7fbd841962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 -- 172.21.15.23:0/3805410740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c009c40 con 0x7fbd84101ab0 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd82d9d700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 0x7fbd841962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:25.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.405+0000 7fbd8259c700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800e9a0 tx=0x7fbd7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.406+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78009790 con 0x7fbd84101ab0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.406+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd740095f0 con 0x7fbd84101ab0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.406+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd78010430 con 0x7fbd84101ab0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.406+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78018680 con 0x7fbd84101ab0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.406+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd84110180 con 0x7fbd84101ab0 2023-11-17T00:00:25.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.407+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8410fff0 con 0x7fbd84101ab0 2023-11-17T00:00:25.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.409+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbd780105a0 con 0x7fbd84101ab0 2023-11-17T00:00:25.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.410+0000 7fbd6bfff700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 0x7fbd6c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:25.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.410+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd78014070 con 0x7fbd84101ab0 2023-11-17T00:00:25.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.410+0000 7fbd82d9d700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 0x7fbd6c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:25.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.411+0000 7fbd82d9d700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 0x7fbd6c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b2e0 tx=0x7fbd74005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:25.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.414+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbd78051730 con 0x7fbd84101ab0 2023-11-17T00:00:25.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.618+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd84110310 con 0x7fbd6c0a11e0 2023-11-17T00:00:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.632+0000 7fbd6bfff700 1 -- 172.21.15.23:0/3805410740 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427244 (secure 0 0 0) 0x7fbd84110310 con 0x7fbd6c0a11e0 2023-11-17T00:00:25.639 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 msgr2=0x7fbd6c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 0x7fbd6c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400b2e0 tx=0x7fbd74005fd0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 msgr2=0x7fbd841967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800e9a0 tx=0x7fbd7800ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 shutdown_connections 2023-11-17T00:00:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fbd6c0a11e0 0x7fbd6c0a3690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd8410a6c0 0x7fbd8419ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd841011a0 0x7fbd841962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 --2- 172.21.15.23:0/3805410740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd84101ab0 0x7fbd841967f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 >> 172.21.15.23:0/3805410740 conn(0x7fbd840761a0 msgr2=0x7fbd84102120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 shutdown_connections 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:25.638+0000 7fbd888f6700 1 -- 172.21.15.23:0/3805410740 wait complete. 2023-11-17T00:00:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-11-17T00:00:26.170 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2023-11-17T00:00:25.079075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654911,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":453,"num_read_kb":342,"num_write":340,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82264,"kb_used_data":13496,"kb_used_omap":0,"kb_used_meta":68672,"kb_avail":1124859560,"statfs":{"total":1151940427776,"available":1151856189440,"internally_reserved":0,"allocated":13819904,"data_stored":7647362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70320128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241211+0000","last_change":"2023-11-16T23:58:46.280372+0000","last_active":"2023-11-16T23:58:51.241211+0000","last_peered":"2023-11-16T23:58:51.241211+0000","last_clean":"2023-11-16T23:58:51.241211+0000","last_became_active":"2023-11-16T23:58:46.280021+0000","last_became_peered":"2023-11-16T23:58:46.280021+0000","last_unstale":"2023-11-16T23:58:51.241211+0000","last_undegraded":"2023-11-16T23:58:51.241211+0000","last_fullsized":"2023-11-16T23:58:51.241211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:37.900152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243114+0000","last_change":"2023-11-16T23:58:46.297046+0000","last_active":"2023-11-16T23:58:51.243114+0000","last_peered":"2023-11-16T23:58:51.243114+0000","last_clean":"2023-11-16T23:58:51.243114+0000","last_became_active":"2023-11-16T23:58:46.282128+0000","last_became_peered":"2023-11-16T23:58:46.282128+0000","last_unstale":"2023-11-16T23:58:51.243114+0000","last_undegraded":"2023-11-16T23:58:51.243114+0000","last_fullsized":"2023-11-16T23:58:51.243114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:56.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242939+0000","last_change":"2023-11-16T23:58:46.283752+0000","last_active":"2023-11-16T23:58:51.242939+0000","last_peered":"2023-11-16T23:58:51.242939+0000","last_clean":"2023-11-16T23:58:51.242939+0000","last_became_active":"2023-11-16T23:58:46.280787+0000","last_became_peered":"2023-11-16T23:58:46.280787+0000","last_unstale":"2023-11-16T23:58:51.242939+0000","last_undegraded":"2023-11-16T23:58:51.242939+0000","last_fullsized":"2023-11-16T23:58:51.242939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:06.203820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974468+0000","last_change":"2023-11-16T23:58:46.282847+0000","last_active":"2023-11-16T23:58:51.974468+0000","last_peered":"2023-11-16T23:58:51.974468+0000","last_clean":"2023-11-16T23:58:51.974468+0000","last_became_active":"2023-11-16T23:58:46.282527+0000","last_became_peered":"2023-11-16T23:58:46.282527+0000","last_unstale":"2023-11-16T23:58:51.974468+0000","last_undegraded":"2023-11-16T23:58:51.974468+0000","last_fullsized":"2023-11-16T23:58:51.974468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:22:15.327419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973484+0000","last_change":"2023-11-16T23:58:46.288154+0000","last_active":"2023-11-16T23:58:51.973484+0000","last_peered":"2023-11-16T23:58:51.973484+0000","last_clean":"2023-11-16T23:58:51.973484+0000","last_became_active":"2023-11-16T23:58:46.287752+0000","last_became_peered":"2023-11-16T23:58:46.287752+0000","last_unstale":"2023-11-16T23:58:51.973484+0000","last_undegraded":"2023-11-16T23:58:51.973484+0000","last_fullsized":"2023-11-16T23:58:51.973484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:17.777171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975139+0000","last_change":"2023-11-16T23:58:46.283177+0000","last_active":"2023-11-16T23:58:51.975139+0000","last_peered":"2023-11-16T23:58:51.975139+0000","last_clean":"2023-11-16T23:58:51.975139+0000","last_became_active":"2023-11-16T23:58:46.282753+0000","last_became_peered":"2023-11-16T23:58:46.282753+0000","last_unstale":"2023-11-16T23:58:51.975139+0000","last_undegraded":"2023-11-16T23:58:51.975139+0000","last_fullsized":"2023-11-16T23:58:51.975139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:40.545830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238882+0000","last_change":"2023-11-16T23:58:46.296002+0000","last_active":"2023-11-16T23:58:51.238882+0000","last_peered":"2023-11-16T23:58:51.238882+0000","last_clean":"2023-11-16T23:58:51.238882+0000","last_became_active":"2023-11-16T23:58:46.295191+0000","last_became_peered":"2023-11-16T23:58:46.295191+0000","last_unstale":"2023-11-16T23:58:51.238882+0000","last_undegraded":"2023-11-16T23:58:51.238882+0000","last_fullsized":"2023-11-16T23:58:51.238882+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:32:43.802637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973904+0000","last_change":"2023-11-16T23:58:46.294824+0000","last_active":"2023-11-16T23:58:51.973904+0000","last_peered":"2023-11-16T23:58:51.973904+0000","last_clean":"2023-11-16T23:58:51.973904+0000","last_became_active":"2023-11-16T23:58:46.294502+0000","last_became_peered":"2023-11-16T23:58:46.294502+0000","last_unstale":"2023-11-16T23:58:51.973904+0000","last_undegraded":"2023-11-16T23:58:51.973904+0000","last_fullsized":"2023-11-16T23:58:51.973904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:47:28.853625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974608+0000","last_change":"2023-11-16T23:58:46.292583+0000","last_active":"2023-11-16T23:58:51.974608+0000","last_peered":"2023-11-16T23:58:51.974608+0000","last_clean":"2023-11-16T23:58:51.974608+0000","last_became_active":"2023-11-16T23:58:46.292089+0000","last_became_peered":"2023-11-16T23:58:46.292089+0000","last_unstale":"2023-11-16T23:58:51.974608+0000","last_undegraded":"2023-11-16T23:58:51.974608+0000","last_fullsized":"2023-11-16T23:58:51.974608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:54.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975325+0000","last_change":"2023-11-16T23:58:46.289410+0000","last_active":"2023-11-16T23:58:51.975325+0000","last_peered":"2023-11-16T23:58:51.975325+0000","last_clean":"2023-11-16T23:58:51.975325+0000","last_became_active":"2023-11-16T23:58:46.288746+0000","last_became_peered":"2023-11-16T23:58:46.288746+0000","last_unstale":"2023-11-16T23:58:51.975325+0000","last_undegraded":"2023-11-16T23:58:51.975325+0000","last_fullsized":"2023-11-16T23:58:51.975325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:39:36.326325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239625+0000","last_change":"2023-11-16T23:58:46.288862+0000","last_active":"2023-11-16T23:58:51.239625+0000","last_peered":"2023-11-16T23:58:51.239625+0000","last_clean":"2023-11-16T23:58:51.239625+0000","last_became_active":"2023-11-16T23:58:46.288046+0000","last_became_peered":"2023-11-16T23:58:46.288046+0000","last_unstale":"2023-11-16T23:58:51.239625+0000","last_undegraded":"2023-11-16T23:58:51.239625+0000","last_fullsized":"2023-11-16T23:58:51.239625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:28:13.350271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974869+0000","last_change":"2023-11-16T23:58:46.283219+0000","last_active":"2023-11-16T23:58:51.974869+0000","last_peered":"2023-11-16T23:58:51.974869+0000","last_clean":"2023-11-16T23:58:51.974869+0000","last_became_active":"2023-11-16T23:58:46.282759+0000","last_became_peered":"2023-11-16T23:58:46.282759+0000","last_unstale":"2023-11-16T23:58:51.974869+0000","last_undegraded":"2023-11-16T23:58:51.974869+0000","last_fullsized":"2023-11-16T23:58:51.974869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:02:48.010586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:46.292139+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:46.291682+0000","last_became_peered":"2023-11-16T23:58:46.291682+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:27:16.238637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242504+0000","last_change":"2023-11-16T23:58:46.293326+0000","last_active":"2023-11-16T23:58:51.242504+0000","last_peered":"2023-11-16T23:58:51.242504+0000","last_clean":"2023-11-16T23:58:51.242504+0000","last_became_active":"2023-11-16T23:58:46.286619+0000","last_became_peered":"2023-11-16T23:58:46.286619+0000","last_unstale":"2023-11-16T23:58:51.242504+0000","last_undegraded":"2023-11-16T23:58:51.242504+0000","last_fullsized":"2023-11-16T23:58:51.242504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:49:41.213234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242574+0000","last_change":"2023-11-16T23:58:46.274095+0000","last_active":"2023-11-16T23:58:51.242574+0000","last_peered":"2023-11-16T23:58:51.242574+0000","last_clean":"2023-11-16T23:58:51.242574+0000","last_became_active":"2023-11-16T23:58:46.272718+0000","last_became_peered":"2023-11-16T23:58:46.272718+0000","last_unstale":"2023-11-16T23:58:51.242574+0000","last_undegraded":"2023-11-16T23:58:51.242574+0000","last_fullsized":"2023-11-16T23:58:51.242574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:04:45.267014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973636+0000","last_change":"2023-11-16T23:58:46.294079+0000","last_active":"2023-11-16T23:58:51.973636+0000","last_peered":"2023-11-16T23:58:51.973636+0000","last_clean":"2023-11-16T23:58:51.973636+0000","last_became_active":"2023-11-16T23:58:46.293738+0000","last_became_peered":"2023-11-16T23:58:46.293738+0000","last_unstale":"2023-11-16T23:58:51.973636+0000","last_undegraded":"2023-11-16T23:58:51.973636+0000","last_fullsized":"2023-11-16T23:58:51.973636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:48.113990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241160+0000","last_change":"2023-11-16T23:58:46.296439+0000","last_active":"2023-11-16T23:58:51.241160+0000","last_peered":"2023-11-16T23:58:51.241160+0000","last_clean":"2023-11-16T23:58:51.241160+0000","last_became_active":"2023-11-16T23:58:46.296002+0000","last_became_peered":"2023-11-16T23:58:46.296002+0000","last_unstale":"2023-11-16T23:58:51.241160+0000","last_undegraded":"2023-11-16T23:58:51.241160+0000","last_fullsized":"2023-11-16T23:58:51.241160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:45.679213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975910+0000","last_change":"2023-11-16T23:58:46.287374+0000","last_active":"2023-11-16T23:58:51.975910+0000","last_peered":"2023-11-16T23:58:51.975910+0000","last_clean":"2023-11-16T23:58:51.975910+0000","last_became_active":"2023-11-16T23:58:46.287044+0000","last_became_peered":"2023-11-16T23:58:46.287044+0000","last_unstale":"2023-11-16T23:58:51.975910+0000","last_undegraded":"2023-11-16T23:58:51.975910+0000","last_fullsized":"2023-11-16T23:58:51.975910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:31:41.509204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416818+0000","last_change":"2023-11-16T23:58:46.284627+0000","last_active":"2023-11-16T23:58:51.416818+0000","last_peered":"2023-11-16T23:58:51.416818+0000","last_clean":"2023-11-16T23:58:51.416818+0000","last_became_active":"2023-11-16T23:58:46.284221+0000","last_became_peered":"2023-11-16T23:58:46.284221+0000","last_unstale":"2023-11-16T23:58:51.416818+0000","last_undegraded":"2023-11-16T23:58:51.416818+0000","last_fullsized":"2023-11-16T23:58:51.416818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:37.717489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416711+0000","last_change":"2023-11-16T23:58:50.742742+0000","last_active":"2023-11-16T23:58:51.416711+0000","last_peered":"2023-11-16T23:58:51.416711+0000","last_clean":"2023-11-16T23:58:51.416711+0000","last_became_active":"2023-11-16T23:58:50.742337+0000","last_became_peered":"2023-11-16T23:58:50.742337+0000","last_unstale":"2023-11-16T23:58:51.416711+0000","last_undegraded":"2023-11-16T23:58:51.416711+0000","last_fullsized":"2023-11-16T23:58:51.416711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:03:40.742862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.717374+0000","last_change":"2023-11-16T23:58:45.714612+0000","last_active":"2023-11-16T23:59:33.717374+0000","last_peered":"2023-11-16T23:59:33.717374+0000","last_clean":"2023-11-16T23:59:33.717374+0000","last_became_active":"2023-11-16T23:58:45.714235+0000","last_became_peered":"2023-11-16T23:58:45.714235+0000","last_unstale":"2023-11-16T23:59:33.717374+0000","last_undegraded":"2023-11-16T23:59:33.717374+0000","last_fullsized":"2023-11-16T23:59:33.717374+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:20:19.287101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974930+0000","last_change":"2023-11-16T23:58:46.294399+0000","last_active":"2023-11-16T23:58:51.974930+0000","last_peered":"2023-11-16T23:58:51.974930+0000","last_clean":"2023-11-16T23:58:51.974930+0000","last_became_active":"2023-11-16T23:58:46.291722+0000","last_became_peered":"2023-11-16T23:58:46.291722+0000","last_unstale":"2023-11-16T23:58:51.974930+0000","last_undegraded":"2023-11-16T23:58:51.974930+0000","last_fullsized":"2023-11-16T23:58:51.974930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:58:25.110448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973733+0000","last_change":"2023-11-16T23:58:50.745764+0000","last_active":"2023-11-16T23:58:51.973733+0000","last_peered":"2023-11-16T23:58:51.973733+0000","last_clean":"2023-11-16T23:58:51.973733+0000","last_became_active":"2023-11-16T23:58:50.745411+0000","last_became_peered":"2023-11-16T23:58:50.745411+0000","last_unstale":"2023-11-16T23:58:51.973733+0000","last_undegraded":"2023-11-16T23:58:51.973733+0000","last_fullsized":"2023-11-16T23:58:51.973733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:04:04.552603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975096+0000","last_change":"2023-11-16T23:58:45.724215+0000","last_active":"2023-11-16T23:58:51.975096+0000","last_peered":"2023-11-16T23:58:51.975096+0000","last_clean":"2023-11-16T23:58:51.975096+0000","last_became_active":"2023-11-16T23:58:45.723866+0000","last_became_peered":"2023-11-16T23:58:45.723866+0000","last_unstale":"2023-11-16T23:58:51.975096+0000","last_undegraded":"2023-11-16T23:58:51.975096+0000","last_fullsized":"2023-11-16T23:58:51.975096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:09.212226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243119+0000","last_change":"2023-11-16T23:58:46.291701+0000","last_active":"2023-11-16T23:58:51.243119+0000","last_peered":"2023-11-16T23:58:51.243119+0000","last_clean":"2023-11-16T23:58:51.243119+0000","last_became_active":"2023-11-16T23:58:46.282612+0000","last_became_peered":"2023-11-16T23:58:46.282612+0000","last_unstale":"2023-11-16T23:58:51.243119+0000","last_undegraded":"2023-11-16T23:58:51.243119+0000","last_fullsized":"2023-11-16T23:58:51.243119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:07:18.169006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975347+0000","last_change":"2023-11-16T23:58:50.731025+0000","last_active":"2023-11-16T23:58:51.975347+0000","last_peered":"2023-11-16T23:58:51.975347+0000","last_clean":"2023-11-16T23:58:51.975347+0000","last_became_active":"2023-11-16T23:58:50.730593+0000","last_became_peered":"2023-11-16T23:58:50.730593+0000","last_unstale":"2023-11-16T23:58:51.975347+0000","last_undegraded":"2023-11-16T23:58:51.975347+0000","last_fullsized":"2023-11-16T23:58:51.975347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:18.213726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715505+0000","last_change":"2023-11-16T23:58:45.716403+0000","last_active":"2023-11-16T23:59:33.715505+0000","last_peered":"2023-11-16T23:59:33.715505+0000","last_clean":"2023-11-16T23:59:33.715505+0000","last_became_active":"2023-11-16T23:58:45.716050+0000","last_became_peered":"2023-11-16T23:58:45.716050+0000","last_unstale":"2023-11-16T23:59:33.715505+0000","last_undegraded":"2023-11-16T23:59:33.715505+0000","last_fullsized":"2023-11-16T23:59:33.715505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:42:55.310961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974544+0000","last_change":"2023-11-16T23:58:46.286460+0000","last_active":"2023-11-16T23:58:51.974544+0000","last_peered":"2023-11-16T23:58:51.974544+0000","last_clean":"2023-11-16T23:58:51.974544+0000","last_became_active":"2023-11-16T23:58:46.286135+0000","last_became_peered":"2023-11-16T23:58:46.286135+0000","last_unstale":"2023-11-16T23:58:51.974544+0000","last_undegraded":"2023-11-16T23:58:51.974544+0000","last_fullsized":"2023-11-16T23:58:51.974544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:15:02.309346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973789+0000","last_change":"2023-11-16T23:58:50.737134+0000","last_active":"2023-11-16T23:58:51.973789+0000","last_peered":"2023-11-16T23:58:51.973789+0000","last_clean":"2023-11-16T23:58:51.973789+0000","last_became_active":"2023-11-16T23:58:50.736489+0000","last_became_peered":"2023-11-16T23:58:50.736489+0000","last_unstale":"2023-11-16T23:58:51.973789+0000","last_undegraded":"2023-11-16T23:58:51.973789+0000","last_fullsized":"2023-11-16T23:58:51.973789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:21:33.961942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973818+0000","last_change":"2023-11-16T23:58:45.713153+0000","last_active":"2023-11-16T23:58:51.973818+0000","last_peered":"2023-11-16T23:58:51.973818+0000","last_clean":"2023-11-16T23:58:51.973818+0000","last_became_active":"2023-11-16T23:58:45.712626+0000","last_became_peered":"2023-11-16T23:58:45.712626+0000","last_unstale":"2023-11-16T23:58:51.973818+0000","last_undegraded":"2023-11-16T23:58:51.973818+0000","last_fullsized":"2023-11-16T23:58:51.973818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:55:36.907838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584145+0000","last_change":"2023-11-16T23:58:46.289729+0000","last_active":"2023-11-16T23:58:51.584145+0000","last_peered":"2023-11-16T23:58:51.584145+0000","last_clean":"2023-11-16T23:58:51.584145+0000","last_became_active":"2023-11-16T23:58:46.284791+0000","last_became_peered":"2023-11-16T23:58:46.284791+0000","last_unstale":"2023-11-16T23:58:51.584145+0000","last_undegraded":"2023-11-16T23:58:51.584145+0000","last_fullsized":"2023-11-16T23:58:51.584145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:13:27.839983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416962+0000","last_change":"2023-11-16T23:58:50.725585+0000","last_active":"2023-11-16T23:58:51.416962+0000","last_peered":"2023-11-16T23:58:51.416962+0000","last_clean":"2023-11-16T23:58:51.416962+0000","last_became_active":"2023-11-16T23:58:50.725188+0000","last_became_peered":"2023-11-16T23:58:50.725188+0000","last_unstale":"2023-11-16T23:58:51.416962+0000","last_undegraded":"2023-11-16T23:58:51.416962+0000","last_fullsized":"2023-11-16T23:58:51.416962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:02:54.017294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:21.781159+0000","last_change":"2023-11-16T23:58:45.715656+0000","last_active":"2023-11-17T00:00:21.781159+0000","last_peered":"2023-11-17T00:00:21.781159+0000","last_clean":"2023-11-17T00:00:21.781159+0000","last_became_active":"2023-11-16T23:58:45.715331+0000","last_became_peered":"2023-11-16T23:58:45.715331+0000","last_unstale":"2023-11-17T00:00:21.781159+0000","last_undegraded":"2023-11-17T00:00:21.781159+0000","last_fullsized":"2023-11-17T00:00:21.781159+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:38:28.590646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973481+0000","last_change":"2023-11-16T23:58:46.294442+0000","last_active":"2023-11-16T23:58:51.973481+0000","last_peered":"2023-11-16T23:58:51.973481+0000","last_clean":"2023-11-16T23:58:51.973481+0000","last_became_active":"2023-11-16T23:58:46.294112+0000","last_became_peered":"2023-11-16T23:58:46.294112+0000","last_unstale":"2023-11-16T23:58:51.973481+0000","last_undegraded":"2023-11-16T23:58:51.973481+0000","last_fullsized":"2023-11-16T23:58:51.973481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:16:37.966411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416678+0000","last_change":"2023-11-16T23:58:50.747104+0000","last_active":"2023-11-16T23:58:51.416678+0000","last_peered":"2023-11-16T23:58:51.416678+0000","last_clean":"2023-11-16T23:58:51.416678+0000","last_became_active":"2023-11-16T23:58:50.746684+0000","last_became_peered":"2023-11-16T23:58:50.746684+0000","last_unstale":"2023-11-16T23:58:51.416678+0000","last_undegraded":"2023-11-16T23:58:51.416678+0000","last_fullsized":"2023-11-16T23:58:51.416678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:23:25.519731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715891+0000","last_change":"2023-11-16T23:58:45.719225+0000","last_active":"2023-11-16T23:59:33.715891+0000","last_peered":"2023-11-16T23:59:33.715891+0000","last_clean":"2023-11-16T23:59:33.715891+0000","last_became_active":"2023-11-16T23:58:45.718815+0000","last_became_peered":"2023-11-16T23:58:45.718815+0000","last_unstale":"2023-11-16T23:59:33.715891+0000","last_undegraded":"2023-11-16T23:59:33.715891+0000","last_fullsized":"2023-11-16T23:59:33.715891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:54:09.302556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583967+0000","last_change":"2023-11-16T23:58:46.292334+0000","last_active":"2023-11-16T23:58:51.583967+0000","last_peered":"2023-11-16T23:58:51.583967+0000","last_clean":"2023-11-16T23:58:51.583967+0000","last_became_active":"2023-11-16T23:58:46.291882+0000","last_became_peered":"2023-11-16T23:58:46.291882+0000","last_unstale":"2023-11-16T23:58:51.583967+0000","last_undegraded":"2023-11-16T23:58:51.583967+0000","last_fullsized":"2023-11-16T23:58:51.583967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:36:17.751014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241151+0000","last_change":"2023-11-16T23:58:50.742737+0000","last_active":"2023-11-16T23:58:51.241151+0000","last_peered":"2023-11-16T23:58:51.241151+0000","last_clean":"2023-11-16T23:58:51.241151+0000","last_became_active":"2023-11-16T23:58:50.742257+0000","last_became_peered":"2023-11-16T23:58:50.742257+0000","last_unstale":"2023-11-16T23:58:51.241151+0000","last_undegraded":"2023-11-16T23:58:51.241151+0000","last_fullsized":"2023-11-16T23:58:51.241151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:47:46.061379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.715391+0000","last_change":"2023-11-16T23:58:45.716537+0000","last_active":"2023-11-16T23:59:33.715391+0000","last_peered":"2023-11-16T23:59:33.715391+0000","last_clean":"2023-11-16T23:59:33.715391+0000","last_became_active":"2023-11-16T23:58:45.716018+0000","last_became_peered":"2023-11-16T23:58:45.716018+0000","last_unstale":"2023-11-16T23:59:33.715391+0000","last_undegraded":"2023-11-16T23:59:33.715391+0000","last_fullsized":"2023-11-16T23:59:33.715391+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:56:57.891968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973828+0000","last_change":"2023-11-16T23:58:46.288713+0000","last_active":"2023-11-16T23:58:51.973828+0000","last_peered":"2023-11-16T23:58:51.973828+0000","last_clean":"2023-11-16T23:58:51.973828+0000","last_became_active":"2023-11-16T23:58:46.288363+0000","last_became_peered":"2023-11-16T23:58:46.288363+0000","last_unstale":"2023-11-16T23:58:51.973828+0000","last_undegraded":"2023-11-16T23:58:51.973828+0000","last_fullsized":"2023-11-16T23:58:51.973828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:48:49.725514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243013+0000","last_change":"2023-11-16T23:58:50.728730+0000","last_active":"2023-11-16T23:58:51.243013+0000","last_peered":"2023-11-16T23:58:51.243013+0000","last_clean":"2023-11-16T23:58:51.243013+0000","last_became_active":"2023-11-16T23:58:50.728164+0000","last_became_peered":"2023-11-16T23:58:50.728164+0000","last_unstale":"2023-11-16T23:58:51.243013+0000","last_undegraded":"2023-11-16T23:58:51.243013+0000","last_fullsized":"2023-11-16T23:58:51.243013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:30:15.323131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583464+0000","last_change":"2023-11-16T23:58:45.713056+0000","last_active":"2023-11-16T23:58:51.583464+0000","last_peered":"2023-11-16T23:58:51.583464+0000","last_clean":"2023-11-16T23:58:51.583464+0000","last_became_active":"2023-11-16T23:58:45.712685+0000","last_became_peered":"2023-11-16T23:58:45.712685+0000","last_unstale":"2023-11-16T23:58:51.583464+0000","last_undegraded":"2023-11-16T23:58:51.583464+0000","last_fullsized":"2023-11-16T23:58:51.583464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:35.811313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974076+0000","last_change":"2023-11-16T23:58:46.291540+0000","last_active":"2023-11-16T23:58:51.974076+0000","last_peered":"2023-11-16T23:58:51.974076+0000","last_clean":"2023-11-16T23:58:51.974076+0000","last_became_active":"2023-11-16T23:58:46.289372+0000","last_became_peered":"2023-11-16T23:58:46.289372+0000","last_unstale":"2023-11-16T23:58:51.974076+0000","last_undegraded":"2023-11-16T23:58:51.974076+0000","last_fullsized":"2023-11-16T23:58:51.974076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:38:19.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974448+0000","last_change":"2023-11-16T23:58:50.729362+0000","last_active":"2023-11-16T23:58:51.974448+0000","last_peered":"2023-11-16T23:58:51.974448+0000","last_clean":"2023-11-16T23:58:51.974448+0000","last_became_active":"2023-11-16T23:58:50.729007+0000","last_became_peered":"2023-11-16T23:58:50.729007+0000","last_unstale":"2023-11-16T23:58:51.974448+0000","last_undegraded":"2023-11-16T23:58:51.974448+0000","last_fullsized":"2023-11-16T23:58:51.974448+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:59.110409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:22.782121+0000","last_change":"2023-11-16T23:58:45.717814+0000","last_active":"2023-11-17T00:00:22.782121+0000","last_peered":"2023-11-17T00:00:22.782121+0000","last_clean":"2023-11-17T00:00:22.782121+0000","last_became_active":"2023-11-16T23:58:45.717494+0000","last_became_peered":"2023-11-16T23:58:45.717494+0000","last_unstale":"2023-11-17T00:00:22.782121+0000","last_undegraded":"2023-11-17T00:00:22.782121+0000","last_fullsized":"2023-11-17T00:00:22.782121+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:33:17.157376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974442+0000","last_change":"2023-11-16T23:58:46.285305+0000","last_active":"2023-11-16T23:58:51.974442+0000","last_peered":"2023-11-16T23:58:51.974442+0000","last_clean":"2023-11-16T23:58:51.974442+0000","last_became_active":"2023-11-16T23:58:46.284563+0000","last_became_peered":"2023-11-16T23:58:46.284563+0000","last_unstale":"2023-11-16T23:58:51.974442+0000","last_undegraded":"2023-11-16T23:58:51.974442+0000","last_fullsized":"2023-11-16T23:58:51.974442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:28:12.936849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973969+0000","last_change":"2023-11-16T23:58:50.740863+0000","last_active":"2023-11-16T23:58:51.973969+0000","last_peered":"2023-11-16T23:58:51.973969+0000","last_clean":"2023-11-16T23:58:51.973969+0000","last_became_active":"2023-11-16T23:58:50.740400+0000","last_became_peered":"2023-11-16T23:58:50.740400+0000","last_unstale":"2023-11-16T23:58:51.973969+0000","last_undegraded":"2023-11-16T23:58:51.973969+0000","last_fullsized":"2023-11-16T23:58:51.973969+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:52:44.691356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733800+0000","last_change":"2023-11-16T23:58:45.715701+0000","last_active":"2023-11-16T23:59:32.733800+0000","last_peered":"2023-11-16T23:59:32.733800+0000","last_clean":"2023-11-16T23:59:32.733800+0000","last_became_active":"2023-11-16T23:58:45.715273+0000","last_became_peered":"2023-11-16T23:58:45.715273+0000","last_unstale":"2023-11-16T23:59:32.733800+0000","last_undegraded":"2023-11-16T23:59:32.733800+0000","last_fullsized":"2023-11-16T23:59:32.733800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:08.019018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.243105+0000","last_change":"2023-11-16T23:58:46.294740+0000","last_active":"2023-11-16T23:58:51.243105+0000","last_peered":"2023-11-16T23:58:51.243105+0000","last_clean":"2023-11-16T23:58:51.243105+0000","last_became_active":"2023-11-16T23:58:46.282681+0000","last_became_peered":"2023-11-16T23:58:46.282681+0000","last_unstale":"2023-11-16T23:58:51.243105+0000","last_undegraded":"2023-11-16T23:58:51.243105+0000","last_fullsized":"2023-11-16T23:58:51.243105+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:44.418038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975559+0000","last_change":"2023-11-16T23:58:50.730244+0000","last_active":"2023-11-16T23:58:51.975559+0000","last_peered":"2023-11-16T23:58:51.975559+0000","last_clean":"2023-11-16T23:58:51.975559+0000","last_became_active":"2023-11-16T23:58:50.729669+0000","last_became_peered":"2023-11-16T23:58:50.729669+0000","last_unstale":"2023-11-16T23:58:51.975559+0000","last_undegraded":"2023-11-16T23:58:51.975559+0000","last_fullsized":"2023-11-16T23:58:51.975559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:47:57.416754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733951+0000","last_change":"2023-11-16T23:58:45.722939+0000","last_active":"2023-11-16T23:59:32.733951+0000","last_peered":"2023-11-16T23:59:32.733951+0000","last_clean":"2023-11-16T23:59:32.733951+0000","last_became_active":"2023-11-16T23:58:45.722559+0000","last_became_peered":"2023-11-16T23:58:45.722559+0000","last_unstale":"2023-11-16T23:59:32.733951+0000","last_undegraded":"2023-11-16T23:59:32.733951+0000","last_fullsized":"2023-11-16T23:59:32.733951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:45:38.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974381+0000","last_change":"2023-11-16T23:58:46.285249+0000","last_active":"2023-11-16T23:58:51.974381+0000","last_peered":"2023-11-16T23:58:51.974381+0000","last_clean":"2023-11-16T23:58:51.974381+0000","last_became_active":"2023-11-16T23:58:46.284815+0000","last_became_peered":"2023-11-16T23:58:46.284815+0000","last_unstale":"2023-11-16T23:58:51.974381+0000","last_undegraded":"2023-11-16T23:58:51.974381+0000","last_fullsized":"2023-11-16T23:58:51.974381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:28.991224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974731+0000","last_change":"2023-11-16T23:58:50.734889+0000","last_active":"2023-11-16T23:58:51.974731+0000","last_peered":"2023-11-16T23:58:51.974731+0000","last_clean":"2023-11-16T23:58:51.974731+0000","last_became_active":"2023-11-16T23:58:50.734461+0000","last_became_peered":"2023-11-16T23:58:50.734461+0000","last_unstale":"2023-11-16T23:58:51.974731+0000","last_undegraded":"2023-11-16T23:58:51.974731+0000","last_fullsized":"2023-11-16T23:58:51.974731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:34:05.742581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":165,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:23.787947+0000","last_change":"2023-11-16T23:58:45.716428+0000","last_active":"2023-11-17T00:00:23.787947+0000","last_peered":"2023-11-17T00:00:23.787947+0000","last_clean":"2023-11-17T00:00:23.787947+0000","last_became_active":"2023-11-16T23:58:45.716076+0000","last_became_peered":"2023-11-16T23:58:45.716076+0000","last_unstale":"2023-11-17T00:00:23.787947+0000","last_undegraded":"2023-11-17T00:00:23.787947+0000","last_fullsized":"2023-11-17T00:00:23.787947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:14:06.840585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60393,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":110,"num_read_kb":113,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973844+0000","last_change":"2023-11-16T23:58:46.282889+0000","last_active":"2023-11-16T23:58:51.973844+0000","last_peered":"2023-11-16T23:58:51.973844+0000","last_clean":"2023-11-16T23:58:51.973844+0000","last_became_active":"2023-11-16T23:58:46.282450+0000","last_became_peered":"2023-11-16T23:58:46.282450+0000","last_unstale":"2023-11-16T23:58:51.973844+0000","last_undegraded":"2023-11-16T23:58:51.973844+0000","last_fullsized":"2023-11-16T23:58:51.973844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:08:09.378619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584460+0000","last_change":"2023-11-16T23:58:50.735438+0000","last_active":"2023-11-16T23:58:51.584460+0000","last_peered":"2023-11-16T23:58:51.584460+0000","last_clean":"2023-11-16T23:58:51.584460+0000","last_became_active":"2023-11-16T23:58:50.735040+0000","last_became_peered":"2023-11-16T23:58:50.735040+0000","last_unstale":"2023-11-16T23:58:51.584460+0000","last_undegraded":"2023-11-16T23:58:51.584460+0000","last_fullsized":"2023-11-16T23:58:51.584460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:12:20.496063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.716638+0000","last_change":"2023-11-16T23:58:45.721161+0000","last_active":"2023-11-16T23:59:33.716638+0000","last_peered":"2023-11-16T23:59:33.716638+0000","last_clean":"2023-11-16T23:59:33.716638+0000","last_became_active":"2023-11-16T23:58:45.720826+0000","last_became_peered":"2023-11-16T23:58:45.720826+0000","last_unstale":"2023-11-16T23:59:33.716638+0000","last_undegraded":"2023-11-16T23:59:33.716638+0000","last_fullsized":"2023-11-16T23:59:33.716638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:33.101318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974461+0000","last_change":"2023-11-16T23:58:46.293152+0000","last_active":"2023-11-16T23:58:51.974461+0000","last_peered":"2023-11-16T23:58:51.974461+0000","last_clean":"2023-11-16T23:58:51.974461+0000","last_became_active":"2023-11-16T23:58:46.292830+0000","last_became_peered":"2023-11-16T23:58:46.292830+0000","last_unstale":"2023-11-16T23:58:51.974461+0000","last_undegraded":"2023-11-16T23:58:51.974461+0000","last_fullsized":"2023-11-16T23:58:51.974461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:06.803488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240824+0000","last_change":"2023-11-16T23:58:50.739327+0000","last_active":"2023-11-16T23:58:51.240824+0000","last_peered":"2023-11-16T23:58:51.240824+0000","last_clean":"2023-11-16T23:58:51.240824+0000","last_became_active":"2023-11-16T23:58:50.739024+0000","last_became_peered":"2023-11-16T23:58:50.739024+0000","last_unstale":"2023-11-16T23:58:51.240824+0000","last_undegraded":"2023-11-16T23:58:51.240824+0000","last_fullsized":"2023-11-16T23:58:51.240824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:08:58.095409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734178+0000","last_change":"2023-11-16T23:58:45.716730+0000","last_active":"2023-11-16T23:59:32.734178+0000","last_peered":"2023-11-16T23:59:32.734178+0000","last_clean":"2023-11-16T23:59:32.734178+0000","last_became_active":"2023-11-16T23:58:45.716380+0000","last_became_peered":"2023-11-16T23:58:45.716380+0000","last_unstale":"2023-11-16T23:59:32.734178+0000","last_undegraded":"2023-11-16T23:59:32.734178+0000","last_fullsized":"2023-11-16T23:59:32.734178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:06:26.975787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239189+0000","last_change":"2023-11-16T23:58:46.290867+0000","last_active":"2023-11-16T23:58:51.239189+0000","last_peered":"2023-11-16T23:58:51.239189+0000","last_clean":"2023-11-16T23:58:51.239189+0000","last_became_active":"2023-11-16T23:58:46.290464+0000","last_became_peered":"2023-11-16T23:58:46.290464+0000","last_unstale":"2023-11-16T23:58:51.239189+0000","last_undegraded":"2023-11-16T23:58:51.239189+0000","last_fullsized":"2023-11-16T23:58:51.239189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:32:23.908610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975901+0000","last_change":"2023-11-16T23:58:50.747045+0000","last_active":"2023-11-16T23:58:51.975901+0000","last_peered":"2023-11-16T23:58:51.975901+0000","last_clean":"2023-11-16T23:58:51.975901+0000","last_became_active":"2023-11-16T23:58:50.743955+0000","last_became_peered":"2023-11-16T23:58:50.743955+0000","last_unstale":"2023-11-16T23:58:51.975901+0000","last_undegraded":"2023-11-16T23:58:51.975901+0000","last_fullsized":"2023-11-16T23:58:51.975901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:54:18.403005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:19.783073+0000","last_change":"2023-11-16T23:58:45.714767+0000","last_active":"2023-11-17T00:00:19.783073+0000","last_peered":"2023-11-17T00:00:19.783073+0000","last_clean":"2023-11-17T00:00:19.783073+0000","last_became_active":"2023-11-16T23:58:45.713817+0000","last_became_peered":"2023-11-16T23:58:45.713817+0000","last_unstale":"2023-11-17T00:00:19.783073+0000","last_undegraded":"2023-11-17T00:00:19.783073+0000","last_fullsized":"2023-11-17T00:00:19.783073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:44:48.694382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238988+0000","last_change":"2023-11-16T23:58:46.295593+0000","last_active":"2023-11-16T23:58:51.238988+0000","last_peered":"2023-11-16T23:58:51.238988+0000","last_clean":"2023-11-16T23:58:51.238988+0000","last_became_active":"2023-11-16T23:58:46.295253+0000","last_became_peered":"2023-11-16T23:58:46.295253+0000","last_unstale":"2023-11-16T23:58:51.238988+0000","last_undegraded":"2023-11-16T23:58:51.238988+0000","last_fullsized":"2023-11-16T23:58:51.238988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:57:11.390297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976616+0000","last_change":"2023-11-16T23:58:50.730370+0000","last_active":"2023-11-16T23:58:51.976616+0000","last_peered":"2023-11-16T23:58:51.976616+0000","last_clean":"2023-11-16T23:58:51.976616+0000","last_became_active":"2023-11-16T23:58:50.729915+0000","last_became_peered":"2023-11-16T23:58:50.729915+0000","last_unstale":"2023-11-16T23:58:51.976616+0000","last_undegraded":"2023-11-16T23:58:51.976616+0000","last_fullsized":"2023-11-16T23:58:51.976616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:40:19.878796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.718774+0000","last_change":"2023-11-16T23:58:45.715792+0000","last_active":"2023-11-16T23:59:33.718774+0000","last_peered":"2023-11-16T23:59:33.718774+0000","last_clean":"2023-11-16T23:59:33.718774+0000","last_became_active":"2023-11-16T23:58:45.715454+0000","last_became_peered":"2023-11-16T23:58:45.715454+0000","last_unstale":"2023-11-16T23:59:33.718774+0000","last_undegraded":"2023-11-16T23:59:33.718774+0000","last_fullsized":"2023-11-16T23:59:33.718774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:51.440176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583583+0000","last_change":"2023-11-16T23:58:46.288249+0000","last_active":"2023-11-16T23:58:51.583583+0000","last_peered":"2023-11-16T23:58:51.583583+0000","last_clean":"2023-11-16T23:58:51.583583+0000","last_became_active":"2023-11-16T23:58:46.287916+0000","last_became_peered":"2023-11-16T23:58:46.287916+0000","last_unstale":"2023-11-16T23:58:51.583583+0000","last_undegraded":"2023-11-16T23:58:51.583583+0000","last_fullsized":"2023-11-16T23:58:51.583583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:34:19.800012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583795+0000","last_change":"2023-11-16T23:58:50.733835+0000","last_active":"2023-11-16T23:58:51.583795+0000","last_peered":"2023-11-16T23:58:51.583795+0000","last_clean":"2023-11-16T23:58:51.583795+0000","last_became_active":"2023-11-16T23:58:50.733373+0000","last_became_peered":"2023-11-16T23:58:50.733373+0000","last_unstale":"2023-11-16T23:58:51.583795+0000","last_undegraded":"2023-11-16T23:58:51.583795+0000","last_fullsized":"2023-11-16T23:58:51.583795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:27:17.246864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.704975+0000","last_change":"2023-11-16T23:58:45.712055+0000","last_active":"2023-11-16T23:59:33.704975+0000","last_peered":"2023-11-16T23:59:33.704975+0000","last_clean":"2023-11-16T23:59:33.704975+0000","last_became_active":"2023-11-16T23:58:45.711618+0000","last_became_peered":"2023-11-16T23:58:45.711618+0000","last_unstale":"2023-11-16T23:59:33.704975+0000","last_undegraded":"2023-11-16T23:59:33.704975+0000","last_fullsized":"2023-11-16T23:59:33.704975+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:37:41.700022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241050+0000","last_change":"2023-11-16T23:58:46.291460+0000","last_active":"2023-11-16T23:58:51.241050+0000","last_peered":"2023-11-16T23:58:51.241050+0000","last_clean":"2023-11-16T23:58:51.241050+0000","last_became_active":"2023-11-16T23:58:46.291144+0000","last_became_peered":"2023-11-16T23:58:46.291144+0000","last_unstale":"2023-11-16T23:58:51.241050+0000","last_undegraded":"2023-11-16T23:58:51.241050+0000","last_fullsized":"2023-11-16T23:58:51.241050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:24:10.888038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973559+0000","last_change":"2023-11-16T23:58:50.927320+0000","last_active":"2023-11-16T23:58:51.973559+0000","last_peered":"2023-11-16T23:58:51.973559+0000","last_clean":"2023-11-16T23:58:51.973559+0000","last_became_active":"2023-11-16T23:58:50.926767+0000","last_became_peered":"2023-11-16T23:58:50.926767+0000","last_unstale":"2023-11-16T23:58:51.973559+0000","last_undegraded":"2023-11-16T23:58:51.973559+0000","last_fullsized":"2023-11-16T23:58:51.973559+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:30:21.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975281+0000","last_change":"2023-11-16T23:58:45.715654+0000","last_active":"2023-11-16T23:58:51.975281+0000","last_peered":"2023-11-16T23:58:51.975281+0000","last_clean":"2023-11-16T23:58:51.975281+0000","last_became_active":"2023-11-16T23:58:45.714899+0000","last_became_peered":"2023-11-16T23:58:45.714899+0000","last_unstale":"2023-11-16T23:58:51.975281+0000","last_undegraded":"2023-11-16T23:58:51.975281+0000","last_fullsized":"2023-11-16T23:58:51.975281+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:53.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974886+0000","last_change":"2023-11-16T23:58:46.291818+0000","last_active":"2023-11-16T23:58:51.974886+0000","last_peered":"2023-11-16T23:58:51.974886+0000","last_clean":"2023-11-16T23:58:51.974886+0000","last_became_active":"2023-11-16T23:58:46.291517+0000","last_became_peered":"2023-11-16T23:58:46.291517+0000","last_unstale":"2023-11-16T23:58:51.974886+0000","last_undegraded":"2023-11-16T23:58:51.974886+0000","last_fullsized":"2023-11-16T23:58:51.974886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:51:34.124036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975632+0000","last_change":"2023-11-16T23:58:50.733546+0000","last_active":"2023-11-16T23:58:51.975632+0000","last_peered":"2023-11-16T23:58:51.975632+0000","last_clean":"2023-11-16T23:58:51.975632+0000","last_became_active":"2023-11-16T23:58:50.733009+0000","last_became_peered":"2023-11-16T23:58:50.733009+0000","last_unstale":"2023-11-16T23:58:51.975632+0000","last_undegraded":"2023-11-16T23:58:51.975632+0000","last_fullsized":"2023-11-16T23:58:51.975632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:37:16.925270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239226+0000","last_change":"2023-11-16T23:58:45.719181+0000","last_active":"2023-11-16T23:58:51.239226+0000","last_peered":"2023-11-16T23:58:51.239226+0000","last_clean":"2023-11-16T23:58:51.239226+0000","last_became_active":"2023-11-16T23:58:45.718826+0000","last_became_peered":"2023-11-16T23:58:45.718826+0000","last_unstale":"2023-11-16T23:58:51.239226+0000","last_undegraded":"2023-11-16T23:58:51.239226+0000","last_fullsized":"2023-11-16T23:58:51.239226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:52:25.180163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975356+0000","last_change":"2023-11-16T23:58:46.283405+0000","last_active":"2023-11-16T23:58:51.975356+0000","last_peered":"2023-11-16T23:58:51.975356+0000","last_clean":"2023-11-16T23:58:51.975356+0000","last_became_active":"2023-11-16T23:58:46.282995+0000","last_became_peered":"2023-11-16T23:58:46.282995+0000","last_unstale":"2023-11-16T23:58:51.975356+0000","last_undegraded":"2023-11-16T23:58:51.975356+0000","last_fullsized":"2023-11-16T23:58:51.975356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:53:28.645169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976551+0000","last_change":"2023-11-16T23:58:50.928220+0000","last_active":"2023-11-16T23:58:51.976551+0000","last_peered":"2023-11-16T23:58:51.976551+0000","last_clean":"2023-11-16T23:58:51.976551+0000","last_became_active":"2023-11-16T23:58:50.927006+0000","last_became_peered":"2023-11-16T23:58:50.927006+0000","last_unstale":"2023-11-16T23:58:51.976551+0000","last_undegraded":"2023-11-16T23:58:51.976551+0000","last_fullsized":"2023-11-16T23:58:51.976551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:09:21.135371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583555+0000","last_change":"2023-11-16T23:58:46.289260+0000","last_active":"2023-11-16T23:58:51.583555+0000","last_peered":"2023-11-16T23:58:51.583555+0000","last_clean":"2023-11-16T23:58:51.583555+0000","last_became_active":"2023-11-16T23:58:46.285041+0000","last_became_peered":"2023-11-16T23:58:46.285041+0000","last_unstale":"2023-11-16T23:58:51.583555+0000","last_undegraded":"2023-11-16T23:58:51.583555+0000","last_fullsized":"2023-11-16T23:58:51.583555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:16.675991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-17T00:00:22.786224+0000","last_change":"2023-11-16T23:58:45.726423+0000","last_active":"2023-11-17T00:00:22.786224+0000","last_peered":"2023-11-17T00:00:22.786224+0000","last_clean":"2023-11-17T00:00:22.786224+0000","last_became_active":"2023-11-16T23:58:45.725996+0000","last_became_peered":"2023-11-16T23:58:45.725996+0000","last_unstale":"2023-11-17T00:00:22.786224+0000","last_undegraded":"2023-11-17T00:00:22.786224+0000","last_fullsized":"2023-11-17T00:00:22.786224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:06:38.058418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:52.683597+0000","last_change":"2023-11-16T23:58:50.730148+0000","last_active":"2023-11-16T23:58:52.683597+0000","last_peered":"2023-11-16T23:58:52.683597+0000","last_clean":"2023-11-16T23:58:52.683597+0000","last_became_active":"2023-11-16T23:58:50.729675+0000","last_became_peered":"2023-11-16T23:58:50.729675+0000","last_unstale":"2023-11-16T23:58:52.683597+0000","last_undegraded":"2023-11-16T23:58:52.683597+0000","last_fullsized":"2023-11-16T23:58:52.683597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:42:47.288350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974143+0000","last_change":"2023-11-16T23:58:46.295437+0000","last_active":"2023-11-16T23:58:51.974143+0000","last_peered":"2023-11-16T23:58:51.974143+0000","last_clean":"2023-11-16T23:58:51.974143+0000","last_became_active":"2023-11-16T23:58:46.291775+0000","last_became_peered":"2023-11-16T23:58:46.291775+0000","last_unstale":"2023-11-16T23:58:51.974143+0000","last_undegraded":"2023-11-16T23:58:51.974143+0000","last_fullsized":"2023-11-16T23:58:51.974143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:59:17.517488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714142+0000","last_change":"2023-11-16T23:58:45.714497+0000","last_active":"2023-11-16T23:59:33.714142+0000","last_peered":"2023-11-16T23:59:33.714142+0000","last_clean":"2023-11-16T23:59:33.714142+0000","last_became_active":"2023-11-16T23:58:45.714167+0000","last_became_peered":"2023-11-16T23:58:45.714167+0000","last_unstale":"2023-11-16T23:59:33.714142+0000","last_undegraded":"2023-11-16T23:59:33.714142+0000","last_fullsized":"2023-11-16T23:59:33.714142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:16:46.839077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.238863+0000","last_change":"2023-11-16T23:57:37.700941+0000","last_active":"2023-11-16T23:58:51.238863+0000","last_peered":"2023-11-16T23:58:51.238863+0000","last_clean":"2023-11-16T23:58:51.238863+0000","last_became_active":"2023-11-16T23:57:37.700379+0000","last_became_peered":"2023-11-16T23:57:37.700379+0000","last_unstale":"2023-11-16T23:58:51.238863+0000","last_undegraded":"2023-11-16T23:58:51.238863+0000","last_fullsized":"2023-11-16T23:58:51.238863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:54:03.597877+0000","last_clean_scrub_stamp":"2023-11-16T23:54:03.597877+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:51:56.505655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584217+0000","last_change":"2023-11-16T23:58:50.927446+0000","last_active":"2023-11-16T23:58:51.584217+0000","last_peered":"2023-11-16T23:58:51.584217+0000","last_clean":"2023-11-16T23:58:51.584217+0000","last_became_active":"2023-11-16T23:58:50.927042+0000","last_became_peered":"2023-11-16T23:58:50.927042+0000","last_unstale":"2023-11-16T23:58:51.584217+0000","last_undegraded":"2023-11-16T23:58:51.584217+0000","last_fullsized":"2023-11-16T23:58:51.584217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:11:06.468189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973431+0000","last_change":"2023-11-16T23:58:46.283611+0000","last_active":"2023-11-16T23:58:51.973431+0000","last_peered":"2023-11-16T23:58:51.973431+0000","last_clean":"2023-11-16T23:58:51.973431+0000","last_became_active":"2023-11-16T23:58:46.281685+0000","last_became_peered":"2023-11-16T23:58:46.281685+0000","last_unstale":"2023-11-16T23:58:51.973431+0000","last_undegraded":"2023-11-16T23:58:51.973431+0000","last_fullsized":"2023-11-16T23:58:51.973431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:28:30.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.714897+0000","last_change":"2023-11-16T23:58:45.716094+0000","last_active":"2023-11-16T23:59:33.714897+0000","last_peered":"2023-11-16T23:59:33.714897+0000","last_clean":"2023-11-16T23:59:33.714897+0000","last_became_active":"2023-11-16T23:58:45.715748+0000","last_became_peered":"2023-11-16T23:58:45.715748+0000","last_unstale":"2023-11-16T23:59:33.714897+0000","last_undegraded":"2023-11-16T23:59:33.714897+0000","last_fullsized":"2023-11-16T23:59:33.714897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:03:48.184073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240803+0000","last_change":"2023-11-16T23:58:50.737281+0000","last_active":"2023-11-16T23:58:51.240803+0000","last_peered":"2023-11-16T23:58:51.240803+0000","last_clean":"2023-11-16T23:58:51.240803+0000","last_became_active":"2023-11-16T23:58:50.736797+0000","last_became_peered":"2023-11-16T23:58:50.736797+0000","last_unstale":"2023-11-16T23:58:51.240803+0000","last_undegraded":"2023-11-16T23:58:51.240803+0000","last_fullsized":"2023-11-16T23:58:51.240803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:39.192922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734179+0000","last_change":"2023-11-16T23:58:45.722935+0000","last_active":"2023-11-16T23:59:32.734179+0000","last_peered":"2023-11-16T23:59:32.734179+0000","last_clean":"2023-11-16T23:59:32.734179+0000","last_became_active":"2023-11-16T23:58:45.722489+0000","last_became_peered":"2023-11-16T23:58:45.722489+0000","last_unstale":"2023-11-16T23:59:32.734179+0000","last_undegraded":"2023-11-16T23:59:32.734179+0000","last_fullsized":"2023-11-16T23:59:32.734179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:58.547593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975877+0000","last_change":"2023-11-16T23:58:46.284611+0000","last_active":"2023-11-16T23:58:51.975877+0000","last_peered":"2023-11-16T23:58:51.975877+0000","last_clean":"2023-11-16T23:58:51.975877+0000","last_became_active":"2023-11-16T23:58:46.283063+0000","last_became_peered":"2023-11-16T23:58:46.283063+0000","last_unstale":"2023-11-16T23:58:51.975877+0000","last_undegraded":"2023-11-16T23:58:51.975877+0000","last_fullsized":"2023-11-16T23:58:51.975877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:48:34.246739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973742+0000","last_change":"2023-11-16T23:58:50.927654+0000","last_active":"2023-11-16T23:58:51.973742+0000","last_peered":"2023-11-16T23:58:51.973742+0000","last_clean":"2023-11-16T23:58:51.973742+0000","last_became_active":"2023-11-16T23:58:50.926936+0000","last_became_peered":"2023-11-16T23:58:50.926936+0000","last_unstale":"2023-11-16T23:58:51.973742+0000","last_undegraded":"2023-11-16T23:58:51.973742+0000","last_fullsized":"2023-11-16T23:58:51.973742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:20.080380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:41.067051+0000","last_change":"2023-11-16T23:58:45.717981+0000","last_active":"2023-11-16T23:59:41.067051+0000","last_peered":"2023-11-16T23:59:41.067051+0000","last_clean":"2023-11-16T23:59:41.067051+0000","last_became_active":"2023-11-16T23:58:45.717702+0000","last_became_peered":"2023-11-16T23:58:45.717702+0000","last_unstale":"2023-11-16T23:59:41.067051+0000","last_undegraded":"2023-11-16T23:59:41.067051+0000","last_fullsized":"2023-11-16T23:59:41.067051+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:32:06.566478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974772+0000","last_change":"2023-11-16T23:58:46.288768+0000","last_active":"2023-11-16T23:58:51.974772+0000","last_peered":"2023-11-16T23:58:51.974772+0000","last_clean":"2023-11-16T23:58:51.974772+0000","last_became_active":"2023-11-16T23:58:46.288430+0000","last_became_peered":"2023-11-16T23:58:46.288430+0000","last_unstale":"2023-11-16T23:58:51.974772+0000","last_undegraded":"2023-11-16T23:58:51.974772+0000","last_fullsized":"2023-11-16T23:58:51.974772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:19:06.093201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416385+0000","last_change":"2023-11-16T23:58:50.724748+0000","last_active":"2023-11-16T23:58:51.416385+0000","last_peered":"2023-11-16T23:58:51.416385+0000","last_clean":"2023-11-16T23:58:51.416385+0000","last_became_active":"2023-11-16T23:58:50.724313+0000","last_became_peered":"2023-11-16T23:58:50.724313+0000","last_unstale":"2023-11-16T23:58:51.416385+0000","last_undegraded":"2023-11-16T23:58:51.416385+0000","last_fullsized":"2023-11-16T23:58:51.416385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:17:21.820364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732623+0000","last_change":"2023-11-16T23:58:45.715563+0000","last_active":"2023-11-16T23:59:32.732623+0000","last_peered":"2023-11-16T23:59:32.732623+0000","last_clean":"2023-11-16T23:59:32.732623+0000","last_became_active":"2023-11-16T23:58:45.715217+0000","last_became_peered":"2023-11-16T23:58:45.715217+0000","last_unstale":"2023-11-16T23:59:32.732623+0000","last_undegraded":"2023-11-16T23:59:32.732623+0000","last_fullsized":"2023-11-16T23:59:32.732623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:41.330842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242993+0000","last_change":"2023-11-16T23:58:46.298305+0000","last_active":"2023-11-16T23:58:51.242993+0000","last_peered":"2023-11-16T23:58:51.242993+0000","last_clean":"2023-11-16T23:58:51.242993+0000","last_became_active":"2023-11-16T23:58:46.297444+0000","last_became_peered":"2023-11-16T23:58:46.297444+0000","last_unstale":"2023-11-16T23:58:51.242993+0000","last_undegraded":"2023-11-16T23:58:51.242993+0000","last_fullsized":"2023-11-16T23:58:51.242993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:06:36.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974077+0000","last_change":"2023-11-16T23:58:50.927915+0000","last_active":"2023-11-16T23:58:51.974077+0000","last_peered":"2023-11-16T23:58:51.974077+0000","last_clean":"2023-11-16T23:58:51.974077+0000","last_became_active":"2023-11-16T23:58:50.927451+0000","last_became_peered":"2023-11-16T23:58:50.927451+0000","last_unstale":"2023-11-16T23:58:51.974077+0000","last_undegraded":"2023-11-16T23:58:51.974077+0000","last_fullsized":"2023-11-16T23:58:51.974077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:21:21.865022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.732946+0000","last_change":"2023-11-16T23:58:45.710870+0000","last_active":"2023-11-16T23:59:32.732946+0000","last_peered":"2023-11-16T23:59:32.732946+0000","last_clean":"2023-11-16T23:59:32.732946+0000","last_became_active":"2023-11-16T23:58:45.710536+0000","last_became_peered":"2023-11-16T23:58:45.710536+0000","last_unstale":"2023-11-16T23:59:32.732946+0000","last_undegraded":"2023-11-16T23:59:32.732946+0000","last_fullsized":"2023-11-16T23:59:32.732946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:10:32.891477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973290+0000","last_change":"2023-11-16T23:58:46.282277+0000","last_active":"2023-11-16T23:58:51.973290+0000","last_peered":"2023-11-16T23:58:51.973290+0000","last_clean":"2023-11-16T23:58:51.973290+0000","last_became_active":"2023-11-16T23:58:46.281913+0000","last_became_peered":"2023-11-16T23:58:46.281913+0000","last_unstale":"2023-11-16T23:58:51.973290+0000","last_undegraded":"2023-11-16T23:58:51.973290+0000","last_fullsized":"2023-11-16T23:58:51.973290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:19:12.760160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976155+0000","last_change":"2023-11-16T23:58:50.740914+0000","last_active":"2023-11-16T23:58:51.976155+0000","last_peered":"2023-11-16T23:58:51.976155+0000","last_clean":"2023-11-16T23:58:51.976155+0000","last_became_active":"2023-11-16T23:58:50.740348+0000","last_became_peered":"2023-11-16T23:58:50.740348+0000","last_unstale":"2023-11-16T23:58:51.976155+0000","last_undegraded":"2023-11-16T23:58:51.976155+0000","last_fullsized":"2023-11-16T23:58:51.976155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:59:30.440820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.734107+0000","last_change":"2023-11-16T23:58:45.723953+0000","last_active":"2023-11-16T23:59:32.734107+0000","last_peered":"2023-11-16T23:59:32.734107+0000","last_clean":"2023-11-16T23:59:32.734107+0000","last_became_active":"2023-11-16T23:58:45.722075+0000","last_became_peered":"2023-11-16T23:58:45.722075+0000","last_unstale":"2023-11-16T23:59:32.734107+0000","last_undegraded":"2023-11-16T23:59:32.734107+0000","last_fullsized":"2023-11-16T23:59:32.734107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:46:29.950815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240548+0000","last_change":"2023-11-16T23:58:46.290530+0000","last_active":"2023-11-16T23:58:51.240548+0000","last_peered":"2023-11-16T23:58:51.240548+0000","last_clean":"2023-11-16T23:58:51.240548+0000","last_became_active":"2023-11-16T23:58:46.290181+0000","last_became_peered":"2023-11-16T23:58:46.290181+0000","last_unstale":"2023-11-16T23:58:51.240548+0000","last_undegraded":"2023-11-16T23:58:51.240548+0000","last_fullsized":"2023-11-16T23:58:51.240548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:39:20.504586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974154+0000","last_change":"2023-11-16T23:58:50.739559+0000","last_active":"2023-11-16T23:58:51.974154+0000","last_peered":"2023-11-16T23:58:51.974154+0000","last_clean":"2023-11-16T23:58:51.974154+0000","last_became_active":"2023-11-16T23:58:50.739156+0000","last_became_peered":"2023-11-16T23:58:50.739156+0000","last_unstale":"2023-11-16T23:58:51.974154+0000","last_undegraded":"2023-11-16T23:58:51.974154+0000","last_fullsized":"2023-11-16T23:58:51.974154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:45.312325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416494+0000","last_change":"2023-11-16T23:58:45.723679+0000","last_active":"2023-11-16T23:58:51.416494+0000","last_peered":"2023-11-16T23:58:51.416494+0000","last_clean":"2023-11-16T23:58:51.416494+0000","last_became_active":"2023-11-16T23:58:45.723340+0000","last_became_peered":"2023-11-16T23:58:45.723340+0000","last_unstale":"2023-11-16T23:58:51.416494+0000","last_undegraded":"2023-11-16T23:58:51.416494+0000","last_fullsized":"2023-11-16T23:58:51.416494+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:28:17.612030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974645+0000","last_change":"2023-11-16T23:58:46.297004+0000","last_active":"2023-11-16T23:58:51.974645+0000","last_peered":"2023-11-16T23:58:51.974645+0000","last_clean":"2023-11-16T23:58:51.974645+0000","last_became_active":"2023-11-16T23:58:46.296622+0000","last_became_peered":"2023-11-16T23:58:46.296622+0000","last_unstale":"2023-11-16T23:58:51.974645+0000","last_undegraded":"2023-11-16T23:58:51.974645+0000","last_fullsized":"2023-11-16T23:58:51.974645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:00:00.033560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974484+0000","last_change":"2023-11-16T23:58:50.927313+0000","last_active":"2023-11-16T23:58:51.974484+0000","last_peered":"2023-11-16T23:58:51.974484+0000","last_clean":"2023-11-16T23:58:51.974484+0000","last_became_active":"2023-11-16T23:58:50.926865+0000","last_became_peered":"2023-11-16T23:58:50.926865+0000","last_unstale":"2023-11-16T23:58:51.974484+0000","last_undegraded":"2023-11-16T23:58:51.974484+0000","last_fullsized":"2023-11-16T23:58:51.974484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:41:57.718515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:32.733211+0000","last_change":"2023-11-16T23:58:45.714188+0000","last_active":"2023-11-16T23:59:32.733211+0000","last_peered":"2023-11-16T23:59:32.733211+0000","last_clean":"2023-11-16T23:59:32.733211+0000","last_became_active":"2023-11-16T23:58:45.713785+0000","last_became_peered":"2023-11-16T23:58:45.713785+0000","last_unstale":"2023-11-16T23:59:32.733211+0000","last_undegraded":"2023-11-16T23:59:32.733211+0000","last_fullsized":"2023-11-16T23:59:32.733211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:49:36.871706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239324+0000","last_change":"2023-11-16T23:58:46.291307+0000","last_active":"2023-11-16T23:58:51.239324+0000","last_peered":"2023-11-16T23:58:51.239324+0000","last_clean":"2023-11-16T23:58:51.239324+0000","last_became_active":"2023-11-16T23:58:46.289946+0000","last_became_peered":"2023-11-16T23:58:46.289946+0000","last_unstale":"2023-11-16T23:58:51.239324+0000","last_undegraded":"2023-11-16T23:58:51.239324+0000","last_fullsized":"2023-11-16T23:58:51.239324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:58:49.392838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.973947+0000","last_change":"2023-11-16T23:58:50.737205+0000","last_active":"2023-11-16T23:58:51.973947+0000","last_peered":"2023-11-16T23:58:51.973947+0000","last_clean":"2023-11-16T23:58:51.973947+0000","last_became_active":"2023-11-16T23:58:50.736756+0000","last_became_peered":"2023-11-16T23:58:50.736756+0000","last_unstale":"2023-11-16T23:58:51.973947+0000","last_undegraded":"2023-11-16T23:58:51.973947+0000","last_fullsized":"2023-11-16T23:58:51.973947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:05:08.988766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:59:33.719115+0000","last_change":"2023-11-16T23:58:45.713979+0000","last_active":"2023-11-16T23:59:33.719115+0000","last_peered":"2023-11-16T23:59:33.719115+0000","last_clean":"2023-11-16T23:59:33.719115+0000","last_became_active":"2023-11-16T23:58:45.713492+0000","last_became_peered":"2023-11-16T23:58:45.713492+0000","last_unstale":"2023-11-16T23:59:33.719115+0000","last_undegraded":"2023-11-16T23:59:33.719115+0000","last_fullsized":"2023-11-16T23:59:33.719115+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T11:17:15.481986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239735+0000","last_change":"2023-11-16T23:58:46.289100+0000","last_active":"2023-11-16T23:58:51.239735+0000","last_peered":"2023-11-16T23:58:51.239735+0000","last_clean":"2023-11-16T23:58:51.239735+0000","last_became_active":"2023-11-16T23:58:46.288506+0000","last_became_peered":"2023-11-16T23:58:46.288506+0000","last_unstale":"2023-11-16T23:58:51.239735+0000","last_undegraded":"2023-11-16T23:58:51.239735+0000","last_fullsized":"2023-11-16T23:58:51.239735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:44:12.486352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242604+0000","last_change":"2023-11-16T23:58:50.741205+0000","last_active":"2023-11-16T23:58:51.242604+0000","last_peered":"2023-11-16T23:58:51.242604+0000","last_clean":"2023-11-16T23:58:51.242604+0000","last_became_active":"2023-11-16T23:58:50.740871+0000","last_became_peered":"2023-11-16T23:58:50.740871+0000","last_unstale":"2023-11-16T23:58:51.242604+0000","last_undegraded":"2023-11-16T23:58:51.242604+0000","last_fullsized":"2023-11-16T23:58:51.242604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:25:17.203133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.584292+0000","last_change":"2023-11-16T23:58:45.723618+0000","last_active":"2023-11-16T23:58:51.584292+0000","last_peered":"2023-11-16T23:58:51.584292+0000","last_clean":"2023-11-16T23:58:51.584292+0000","last_became_active":"2023-11-16T23:58:45.723266+0000","last_became_peered":"2023-11-16T23:58:45.723266+0000","last_unstale":"2023-11-16T23:58:51.584292+0000","last_undegraded":"2023-11-16T23:58:51.584292+0000","last_fullsized":"2023-11-16T23:58:51.584292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:20:44.206104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239349+0000","last_change":"2023-11-16T23:58:46.295295+0000","last_active":"2023-11-16T23:58:51.239349+0000","last_peered":"2023-11-16T23:58:51.239349+0000","last_clean":"2023-11-16T23:58:51.239349+0000","last_became_active":"2023-11-16T23:58:46.294801+0000","last_became_peered":"2023-11-16T23:58:46.294801+0000","last_unstale":"2023-11-16T23:58:51.239349+0000","last_undegraded":"2023-11-16T23:58:51.239349+0000","last_fullsized":"2023-11-16T23:58:51.239349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:17:00.523212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583352+0000","last_change":"2023-11-16T23:58:50.725856+0000","last_active":"2023-11-16T23:58:51.583352+0000","last_peered":"2023-11-16T23:58:51.583352+0000","last_clean":"2023-11-16T23:58:51.583352+0000","last_became_active":"2023-11-16T23:58:50.725453+0000","last_became_peered":"2023-11-16T23:58:50.725453+0000","last_unstale":"2023-11-16T23:58:51.583352+0000","last_undegraded":"2023-11-16T23:58:51.583352+0000","last_fullsized":"2023-11-16T23:58:51.583352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:49.702045+0000","last_clean_scrub_stamp":"2023-11-16T23:58:49.702045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:32:17.547808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976506+0000","last_change":"2023-11-16T23:58:45.718226+0000","last_active":"2023-11-16T23:58:51.976506+0000","last_peered":"2023-11-16T23:58:51.976506+0000","last_clean":"2023-11-16T23:58:51.976506+0000","last_became_active":"2023-11-16T23:58:45.717893+0000","last_became_peered":"2023-11-16T23:58:45.717893+0000","last_unstale":"2023-11-16T23:58:51.976506+0000","last_undegraded":"2023-11-16T23:58:51.976506+0000","last_fullsized":"2023-11-16T23:58:51.976506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:44.683955+0000","last_clean_scrub_stamp":"2023-11-16T23:58:44.683955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:04:01.114364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.239740+0000","last_change":"2023-11-16T23:58:46.289906+0000","last_active":"2023-11-16T23:58:51.239740+0000","last_peered":"2023-11-16T23:58:51.239740+0000","last_clean":"2023-11-16T23:58:51.239740+0000","last_became_active":"2023-11-16T23:58:46.289266+0000","last_became_peered":"2023-11-16T23:58:46.289266+0000","last_unstale":"2023-11-16T23:58:51.239740+0000","last_undegraded":"2023-11-16T23:58:51.239740+0000","last_fullsized":"2023-11-16T23:58:51.239740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:54:03.586113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240620+0000","last_change":"2023-11-16T23:58:46.292220+0000","last_active":"2023-11-16T23:58:51.240620+0000","last_peered":"2023-11-16T23:58:51.240620+0000","last_clean":"2023-11-16T23:58:51.240620+0000","last_became_active":"2023-11-16T23:58:46.291840+0000","last_became_peered":"2023-11-16T23:58:46.291840+0000","last_unstale":"2023-11-16T23:58:51.240620+0000","last_undegraded":"2023-11-16T23:58:51.240620+0000","last_fullsized":"2023-11-16T23:58:51.240620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:18:35.280653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416702+0000","last_change":"2023-11-16T23:58:46.287601+0000","last_active":"2023-11-16T23:58:51.416702+0000","last_peered":"2023-11-16T23:58:51.416702+0000","last_clean":"2023-11-16T23:58:51.416702+0000","last_became_active":"2023-11-16T23:58:46.284738+0000","last_became_peered":"2023-11-16T23:58:46.284738+0000","last_unstale":"2023-11-16T23:58:51.416702+0000","last_undegraded":"2023-11-16T23:58:51.416702+0000","last_fullsized":"2023-11-16T23:58:51.416702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:49.913458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416384+0000","last_change":"2023-11-16T23:58:46.288601+0000","last_active":"2023-11-16T23:58:51.416384+0000","last_peered":"2023-11-16T23:58:51.416384+0000","last_clean":"2023-11-16T23:58:51.416384+0000","last_became_active":"2023-11-16T23:58:46.288266+0000","last_became_peered":"2023-11-16T23:58:46.288266+0000","last_unstale":"2023-11-16T23:58:51.416384+0000","last_undegraded":"2023-11-16T23:58:51.416384+0000","last_fullsized":"2023-11-16T23:58:51.416384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:41:18.659483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.242778+0000","last_change":"2023-11-16T23:58:46.286495+0000","last_active":"2023-11-16T23:58:51.242778+0000","last_peered":"2023-11-16T23:58:51.242778+0000","last_clean":"2023-11-16T23:58:51.242778+0000","last_became_active":"2023-11-16T23:58:46.281157+0000","last_became_peered":"2023-11-16T23:58:46.281157+0000","last_unstale":"2023-11-16T23:58:51.242778+0000","last_undegraded":"2023-11-16T23:58:51.242778+0000","last_fullsized":"2023-11-16T23:58:51.242778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T05:08:02.565515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.976001+0000","last_change":"2023-11-16T23:58:46.282373+0000","last_active":"2023-11-16T23:58:51.976001+0000","last_peered":"2023-11-16T23:58:51.976001+0000","last_clean":"2023-11-16T23:58:51.976001+0000","last_became_active":"2023-11-16T23:58:46.282028+0000","last_became_peered":"2023-11-16T23:58:46.282028+0000","last_unstale":"2023-11-16T23:58:51.976001+0000","last_undegraded":"2023-11-16T23:58:51.976001+0000","last_fullsized":"2023-11-16T23:58:51.976001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:29:48.640544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.241073+0000","last_change":"2023-11-16T23:58:46.285528+0000","last_active":"2023-11-16T23:58:51.241073+0000","last_peered":"2023-11-16T23:58:51.241073+0000","last_clean":"2023-11-16T23:58:51.241073+0000","last_became_active":"2023-11-16T23:58:46.285227+0000","last_became_peered":"2023-11-16T23:58:46.285227+0000","last_unstale":"2023-11-16T23:58:51.241073+0000","last_undegraded":"2023-11-16T23:58:51.241073+0000","last_fullsized":"2023-11-16T23:58:51.241073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T06:45:11.259431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.416982+0000","last_change":"2023-11-16T23:58:46.283514+0000","last_active":"2023-11-16T23:58:51.416982+0000","last_peered":"2023-11-16T23:58:51.416982+0000","last_clean":"2023-11-16T23:58:51.416982+0000","last_became_active":"2023-11-16T23:58:46.283026+0000","last_became_peered":"2023-11-16T23:58:46.283026+0000","last_unstale":"2023-11-16T23:58:51.416982+0000","last_undegraded":"2023-11-16T23:58:51.416982+0000","last_fullsized":"2023-11-16T23:58:51.416982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:39:14.255094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974891+0000","last_change":"2023-11-16T23:58:46.283334+0000","last_active":"2023-11-16T23:58:51.974891+0000","last_peered":"2023-11-16T23:58:51.974891+0000","last_clean":"2023-11-16T23:58:51.974891+0000","last_became_active":"2023-11-16T23:58:46.283012+0000","last_became_peered":"2023-11-16T23:58:46.283012+0000","last_unstale":"2023-11-16T23:58:51.974891+0000","last_undegraded":"2023-11-16T23:58:51.974891+0000","last_fullsized":"2023-11-16T23:58:51.974891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:44.161087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.583457+0000","last_change":"2023-11-16T23:58:46.292720+0000","last_active":"2023-11-16T23:58:51.583457+0000","last_peered":"2023-11-16T23:58:51.583457+0000","last_clean":"2023-11-16T23:58:51.583457+0000","last_became_active":"2023-11-16T23:58:46.292109+0000","last_became_peered":"2023-11-16T23:58:46.292109+0000","last_unstale":"2023-11-16T23:58:51.583457+0000","last_undegraded":"2023-11-16T23:58:51.583457+0000","last_fullsized":"2023-11-16T23:58:51.583457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T09:41:12.067371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.974297+0000","last_change":"2023-11-16T23:58:46.279919+0000","last_active":"2023-11-16T23:58:51.974297+0000","last_peered":"2023-11-16T23:58:51.974297+0000","last_clean":"2023-11-16T23:58:51.974297+0000","last_became_active":"2023-11-16T23:58:46.279601+0000","last_became_peered":"2023-11-16T23:58:46.279601+0000","last_unstale":"2023-11-16T23:58:51.974297+0000","last_undegraded":"2023-11-16T23:58:51.974297+0000","last_fullsized":"2023-11-16T23:58:51.974297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:28:22.080722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975063+0000","last_change":"2023-11-16T23:58:46.293091+0000","last_active":"2023-11-16T23:58:51.975063+0000","last_peered":"2023-11-16T23:58:51.975063+0000","last_clean":"2023-11-16T23:58:51.975063+0000","last_became_active":"2023-11-16T23:58:46.292728+0000","last_became_peered":"2023-11-16T23:58:46.292728+0000","last_unstale":"2023-11-16T23:58:51.975063+0000","last_undegraded":"2023-11-16T23:58:51.975063+0000","last_fullsized":"2023-11-16T23:58:51.975063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T07:57:24.095879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.975716+0000","last_change":"2023-11-16T23:58:46.281708+0000","last_active":"2023-11-16T23:58:51.975716+0000","last_peered":"2023-11-16T23:58:51.975716+0000","last_clean":"2023-11-16T23:58:51.975716+0000","last_became_active":"2023-11-16T23:58:46.281335+0000","last_became_peered":"2023-11-16T23:58:46.281335+0000","last_unstale":"2023-11-16T23:58:51.975716+0000","last_undegraded":"2023-11-16T23:58:51.975716+0000","last_fullsized":"2023-11-16T23:58:51.975716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T08:46:24.208579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-11-16T23:58:51.240805+0000","last_change":"2023-11-16T23:58:46.286046+0000","last_active":"2023-11-16T23:58:51.240805+0000","last_peered":"2023-11-16T23:58:51.240805+0000","last_clean":"2023-11-16T23:58:51.240805+0000","last_became_active":"2023-11-16T23:58:46.285702+0000","last_became_peered":"2023-11-16T23:58:46.285702+0000","last_unstale":"2023-11-16T23:58:51.240805+0000","last_undegraded":"2023-11-16T23:58:51.240805+0000","last_fullsized":"2023-11-16T23:58:51.240805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T23:58:45.690702+0000","last_clean_scrub_stamp":"2023-11-16T23:58:45.690702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T10:15:25.406768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64297,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":327,"num_read_kb":233,"num_write":157,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":192891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":79,"seq":339302416415,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":974848,"data_stored":475644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86199999999999999,"5min":0.874,"15min":0.874},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.49399999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45800000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.56000000000000005,"5min":0.433,"15min":0.433},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85399999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.496},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79700000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.45200000000000001}]},{"osd":6,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70699999999999996},"min":{"1min":0.56100000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69699999999999995}]},{"osd":7,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.71999999999999997}]},{"osd":8,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60399999999999998}]},{"osd":9,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76900000000000002,"15min":0.76600000000000001},"min":{"1min":0.65000000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.94899999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77400000000000002,"15min":0.76900000000000002},"min":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.96099999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.73599999999999999}]},{"osd":10,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.86899999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.751},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.752,"15min":0.751},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.90300000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78100000000000003}]}]},{"osd":10,"up_from":71,"seq":304942678052,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7068,"kb_used_data":1620,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738084,"statfs":{"total":95995035648,"available":95987798016,"internally_reserved":0,"allocated":1658880,"data_stored":1122926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.47299999999999998,"15min":0.47699999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79700000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.501}]},{"osd":3,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79600000000000004,"5min":0.877,"15min":0.877},"last":0.748},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.36299999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.754,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.70299999999999996},"min":{"1min":0.54200000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.85399999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.69799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83799999999999997,"5min":1.103,"15min":1.103},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91300000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.434},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71599999999999997,"15min":0.71999999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.80100000000000005,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.94099999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60499999999999998}]},{"osd":9,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.97799999999999998,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.84799999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.78500000000000003}]},{"osd":11,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90500000000000003,"5min":1.034,"15min":1.034},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79600000000000004,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.89000000000000001,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.71899999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841580,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6292,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738860,"statfs":{"total":95995035648,"available":95988592640,"internally_reserved":0,"allocated":995328,"data_stored":479976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.728,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.58599999999999997,"15min":0.61399999999999999},"min":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":1.5469999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.8320000000000001,"5min":1.899,"15min":1.899},"last":0.72999999999999998}]},{"osd":2,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.78200000000000003,"15min":0.80700000000000005},"min":{"1min":0.437,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.76500000000000001,"15min":0.78100000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.68999999999999995}]},{"osd":3,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.65900000000000003},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.7030000000000001,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.80100000000000005,"15min":0.84299999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.794,"5min":2.056,"15min":2.056},"last":0.627}]},{"osd":4,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.81299999999999994,"15min":0.85899999999999999},"min":{"1min":0.48799999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.82199999999999995},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6299999999999999,"5min":2.677,"15min":2.677},"last":0.60199999999999998}]},{"osd":5,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.872,"15min":0.91900000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2310000000000001,"5min":1.329,"15min":1.329},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.875,"15min":0.88300000000000001},"min":{"1min":0.63,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.5860000000000001,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.51800000000000002}]},{"osd":6,"last update":"Thu Nov 16 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.86299999999999999}]},{"osd":7,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.97099999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.6699999999999999,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.98099999999999998,"15min":1.0249999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9399999999999999,"5min":1.98,"15min":1.98},"last":0.96599999999999997}]},{"osd":8,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.98099999999999998,"15min":1.0269999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.98399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.008,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.71499999999999997}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.016},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.874,"15min":0.878},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0649999999999999,"5min":1.141,"15min":1.141},"last":0.90800000000000003}]},{"osd":11,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.78800000000000003}]}]},{"osd":8,"up_from":58,"seq":249108103217,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6480,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738672,"statfs":{"total":95995035648,"available":95988400128,"internally_reserved":0,"allocated":991232,"data_stored":475800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.879,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57099999999999995,"15min":0.58099999999999996},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.752}]},{"osd":1,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59999999999999998,"15min":0.59899999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.42199999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89600000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.97899999999999998}]},{"osd":2,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47899999999999998,"15min":0.49299999999999999},"min":{"1min":0.312,"5min":0.222,"15min":0.222},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68200000000000005,"15min":0.66800000000000004},"min":{"1min":0.42599999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.86199999999999999,"5min":4.5149999999999997,"15min":4.5149999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84499999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.622}]},{"osd":4,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.81299999999999994}]},{"osd":5,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54800000000000004,"15min":0.56399999999999995},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83499999999999996,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60299999999999998,"15min":0.63500000000000001},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80700000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.91700000000000004}]},{"osd":6,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.72799999999999998},"min":{"1min":0.45100000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76100000000000001,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92000000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68600000000000005,"15min":0.69799999999999995},"min":{"1min":0.47499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.081,"15min":1.081},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.73599999999999999},"min":{"1min":0.47899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.82899999999999996}]},{"osd":9,"last update":"Fri Nov 17 00:00:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.77000000000000002,"15min":0.77100000000000002},"min":{"1min":0.63100000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.04,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.78800000000000003,"15min":0.77200000000000002},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.997,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.046}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98399999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.141,"5min":1.145,"15min":1.145},"last":0.77400000000000002}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":1.022},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.95399999999999996}]}]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7124,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738028,"statfs":{"total":95995035648,"available":95987740672,"internally_reserved":0,"allocated":1585152,"data_stored":1071612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35299999999999998}]},{"osd":1,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41799999999999998,"15min":0.40999999999999998},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.44500000000000001,"15min":0.435},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.70999999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.372}]},{"osd":2,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56899999999999995,"15min":0.54700000000000004},"min":{"1min":0.48699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80000000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.622}]},{"osd":3,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56200000000000006},"min":{"1min":0.45100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78400000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.59299999999999997}]},{"osd":4,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.35099999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.77700000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.505,"15min":0.505},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40999999999999998}]},{"osd":5,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.51200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.60499999999999998},"min":{"1min":0.45600000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Nov 17 00:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97699999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.67500000000000004},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92500000000000004,"5min":1.069,"15min":1.069},"last":0.57899999999999996}]},{"osd":8,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64100000000000001,"15min":0.61399999999999999},"min":{"1min":0.499,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90300000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.64600000000000002,"15min":0.61799999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86299999999999999,"5min":1.071,"15min":1.071},"last":0.67900000000000005}]},{"osd":9,"last update":"Fri Nov 17 00:00:04 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92300000000000004,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.61099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0109999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.72999999999999998}]},{"osd":10,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.51700000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.82199999999999995,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.76600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6396,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738756,"statfs":{"total":95995035648,"available":95988486144,"internally_reserved":0,"allocated":970752,"data_stored":471056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.52500000000000002,"15min":0.501},"min":{"1min":0.38400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.375,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.66500000000000004,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.60999999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67800000000000005,"15min":0.65100000000000002},"min":{"1min":0.52500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.57099999999999995,"5min":0.432,"15min":0.432},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":0.752}]},{"osd":3,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.52200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.746,"15min":0.76000000000000001},"min":{"1min":0.51300000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":2.6240000000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.80400000000000005,"15min":0.79700000000000004},"min":{"1min":0.67500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.78600000000000003,"15min":0.76600000000000001},"min":{"1min":0.56100000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Nov 17 00:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.624,"5min":0.443,"15min":0.443},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.81499999999999995,"15min":0.78300000000000003},"min":{"1min":0.66000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.85399999999999998}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.86499999999999999,"15min":0.86799999999999999},"min":{"1min":0.626,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.86499999999999999,"15min":0.81899999999999995},"min":{"1min":0.65100000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.256,"5min":5.5839999999999996,"15min":5.5839999999999996},"last":0.58899999999999997}]},{"osd":9,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.94299999999999995,"15min":0.93500000000000005},"min":{"1min":0.58399999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":0.68799999999999994}]},{"osd":10,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93700000000000006,"15min":0.93500000000000005},"min":{"1min":0.66900000000000004,"5min":0.63,"15min":0.63},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.90500000000000003,"15min":0.90100000000000002},"min":{"1min":0.70599999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.3180000000000001,"5min":1.351,"15min":1.351},"last":0.76500000000000001}]},{"osd":11,"last update":"Thu Nov 16 23:59:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.73699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":1036288,"data_stored":531359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.51600000000000001},"min":{"1min":0.41099999999999998,"5min":0.314,"15min":0.30099999999999999},"max":{"1min":0.85599999999999998,"5min":0.998,"15min":0.998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59899999999999998,"15min":0.51900000000000002},"min":{"1min":0.46400000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84299999999999997,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.75800000000000001}]},{"osd":2,"last update":"Fri Nov 17 00:00:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59499999999999997,"15min":0.55500000000000005},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":0.878,"15min":0.878},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.501,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83299999999999996,"5min":1.054,"15min":1.054},"last":0.72299999999999998}]},{"osd":3,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.79900000000000004}]},{"osd":4,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.71199999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.40799999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.628,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.55600000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.877,"5min":1.103,"15min":1.103},"last":0.82399999999999995}]},{"osd":6,"last update":"Thu Nov 16 23:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96499999999999997,"5min":1.159,"15min":1.159},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.89500000000000002,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Nov 16 23:59:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57999999999999996,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.58499999999999996}]},{"osd":8,"last update":"Thu Nov 16 23:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68700000000000006,"15min":0.64400000000000002},"min":{"1min":0.54100000000000004,"5min":0.37,"15min":0.37},"max":{"1min":0.94199999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68999999999999995,"15min":0.65000000000000002},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94299999999999995,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.873}]},{"osd":9,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.627,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93100000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.57599999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0049999999999999,"5min":1.173,"15min":1.173},"last":0.85899999999999999}]},{"osd":10,"last update":"Thu Nov 16 23:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78700000000000003,"15min":0.79100000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0149999999999999,"5min":1.133,"15min":1.133},"last":0.91700000000000004}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.95199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":472444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61199999999999999,"15min":0.55900000000000005},"min":{"1min":0.46100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.64400000000000002,"15min":0.57999999999999996},"min":{"1min":0.48099999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.70499999999999996}]},{"osd":2,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.625,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.29099999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.621},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.61599999999999999},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.35699999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.51200000000000001}]},{"osd":3,"last update":"Thu Nov 16 23:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.78800000000000003}]},{"osd":4,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.71699999999999997,"15min":0.63700000000000001},"min":{"1min":0.61399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.436,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Nov 16 23:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.61199999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.496},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67200000000000004,"15min":0.65500000000000003},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.45400000000000001}]},{"osd":6,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68700000000000006,"15min":0.67900000000000005},"min":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.79300000000000004,"15min":0.72799999999999998},"min":{"1min":0.59199999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Nov 16 23:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.77100000000000002}]},{"osd":8,"last update":"Thu Nov 16 23:59:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.874,"15min":0.872},"min":{"1min":0.55700000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.175,"5min":1.276,"15min":1.276},"last":0.57499999999999996}]},{"osd":9,"last update":"Fri Nov 17 00:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86499999999999999,"15min":0.879},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93500000000000005,"15min":0.93300000000000005},"min":{"1min":0.67800000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.194,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.752}]},{"osd":10,"last update":"Thu Nov 16 23:59:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.90200000000000002,"15min":0.874},"min":{"1min":0.68300000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.877},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.85299999999999998,"15min":0.83899999999999997},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.80200000000000005}]},{"osd":11,"last update":"Fri Nov 17 00:00:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.872,"15min":0.85099999999999998},"min":{"1min":0.68899999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.94699999999999995,"15min":0.93899999999999995},"min":{"1min":0.79200000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3160000000000001,"5min":1.385,"15min":1.385},"last":0.59999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7572,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737580,"statfs":{"total":95995035648,"available":95987281920,"internally_reserved":0,"allocated":1060864,"data_stored":533051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 00:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78200000000000003,"5min":1.675,"15min":1.675},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Nov 17 00:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.55100000000000005,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.621}]},{"osd":3,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Nov 16 23:59:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Nov 16 23:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Nov 16 23:59:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.57299999999999995,"5min":0.37,"15min":0.37},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.624,"15min":0.60199999999999998},"min":{"1min":0.56200000000000006,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.872,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72699999999999998}]},{"osd":7,"last update":"Thu Nov 16 23:59:49 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65500000000000003}]},{"osd":8,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.64800000000000002,"15min":0.63800000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.91300000000000003}]},{"osd":9,"last update":"Fri Nov 17 00:00:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73499999999999999,"15min":0.71699999999999997},"min":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96599999999999997,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.74199999999999999,"15min":0.71799999999999997},"min":{"1min":0.65000000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.93600000000000005,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.83699999999999997}]},{"osd":10,"last update":"Thu Nov 16 23:59:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.65900000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.872},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.88200000000000001}]},{"osd":11,"last update":"Thu Nov 16 23:59:45 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.93600000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6660,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738492,"statfs":{"total":95995035648,"available":95988215808,"internally_reserved":0,"allocated":978944,"data_stored":470919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69699999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.44900000000000001,"15min":0.40699999999999997},"min":{"1min":0.34399999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.51900000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.65500000000000003,"15min":0.59899999999999998},"min":{"1min":0.55100000000000005,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94799999999999995,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Nov 16 23:59:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71399999999999997,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.88500000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79400000000000004}]},{"osd":4,"last update":"Fri Nov 17 00:00:13 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.72099999999999997,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.68899999999999995,"15min":0.629},"min":{"1min":0.52300000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.62}]},{"osd":5,"last update":"Thu Nov 16 23:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69599999999999995,"15min":0.63900000000000001},"min":{"1min":0.51400000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.1220000000000001,"5min":1.492,"15min":1.492},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68999999999999995,"15min":0.64600000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.212,"15min":1.212},"last":0.76500000000000001}]},{"osd":6,"last update":"Thu Nov 16 23:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60599999999999998,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.016,"5min":1.093,"15min":1.093},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.69599999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.214,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Nov 17 00:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.84599999999999997,"15min":0.80500000000000005},"min":{"1min":0.59999999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.79700000000000004,"15min":0.76500000000000001},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Nov 16 23:58:38 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.90600000000000003,"15min":0.86799999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":7.173,"5min":7.173,"15min":7.173},"last":0.84499999999999997}]},{"osd":9,"last update":"Fri Nov 17 00:00:11 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.747},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.97199999999999998,"15min":1.0049999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.377,"5min":1.617,"15min":1.617},"last":0.84599999999999997}]},{"osd":10,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.212,"15min":1.2609999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.212,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.89500000000000002,"15min":0.90300000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.1919999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Nov 16 23:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.89400000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":983040,"data_stored":475658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40500000000000003,"15min":0.36699999999999999},"min":{"1min":0.35599999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48599999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.40200000000000002,"15min":0.372},"min":{"1min":0.34999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.45100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57199999999999995,"15min":0.54200000000000004},"min":{"1min":0.498,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67300000000000004,"5min":0.872,"15min":0.872},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Nov 17 00:00:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64800000000000002,"15min":0.59299999999999997},"min":{"1min":0.54400000000000004,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76700000000000002,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65000000000000002,"15min":0.621},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78700000000000003,"5min":1.105,"15min":1.105},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Nov 16 23:59:31 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Nov 17 00:00:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.60299999999999998},"min":{"1min":0.495,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93400000000000005,"5min":0.997,"15min":0.997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Nov 16 23:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.73899999999999999}]},{"osd":8,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.52400000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.57699999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83899999999999997}]},{"osd":9,"last update":"Fri Nov 17 00:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.72699999999999998},"min":{"1min":0.52800000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.97999999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.64100000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.995,"5min":1.05,"15min":1.05},"last":0.90100000000000002}]},{"osd":10,"last update":"Thu Nov 16 23:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.51100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88900000000000001,"5min":1.083,"15min":1.083},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78700000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.75800000000000001}]},{"osd":11,"last update":"Thu Nov 16 23:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.82799999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":1589248,"data_stored":1066917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52000000000000002},"min":{"1min":0.42199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.753},"last":0.624},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72699999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.753,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.73899999999999999}]},{"osd":2,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40400000000000003,"15min":0.38800000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55700000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.38700000000000001},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57799999999999996,"15min":0.55200000000000005},"min":{"1min":0.47099999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.65600000000000003}]},{"osd":4,"last update":"Thu Nov 16 23:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.55300000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81299999999999994,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.44,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.875}]},{"osd":6,"last update":"Thu Nov 16 23:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63,"15min":0.62},"min":{"1min":0.443,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.752},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Nov 17 00:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65100000000000002,"15min":0.63},"min":{"1min":0.56399999999999995,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83799999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.502,"5min":0.439,"15min":0.439},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.72599999999999998}]},{"osd":8,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.45500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.86199999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.90100000000000002}]},{"osd":9,"last update":"Fri Nov 17 00:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.57699999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71799999999999997,"15min":0.70899999999999996},"min":{"1min":0.56899999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.91200000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.82399999999999995}]},{"osd":10,"last update":"Thu Nov 16 23:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.66900000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.72399999999999998,"15min":0.71699999999999997},"min":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.81599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T00:00:29.866 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-17T00:00:29.866 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-17T00:00:29.867 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-17T00:00:29.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph health --format=json 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:26 smithi156 ceph-mon[131105]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:27 smithi156 ceph-mon[131105]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:28 smithi156 ceph-mon[131105]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:26 smithi173 ceph-mon[131534]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:27 smithi173 ceph-mon[131534]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:28 smithi173 ceph-mon[131534]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:26 smithi023 ceph-mon[97824]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:27 smithi023 ceph-mon[97824]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T00:00:29.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:28 smithi023 ceph-mon[97824]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:30.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:29 smithi023 ceph-mon[97824]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:29 smithi173 ceph-mon[131534]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:29 smithi156 ceph-mon[131105]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:31.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-11-17T00:00:31.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-11-17T00:00:31.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-17T00:00:31.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-11-17T00:00:31.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-11-17T00:00:31.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-11-17T00:00:31.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-11-17T00:00:31.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-11-17T00:00:32.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: osdmap e89: 12 total, 12 up, 12 in 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-17T00:00:32.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:32 smithi023 ceph-mon[97824]: osdmap e90: 12 total, 12 up, 12 in 2023-11-17T00:00:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 -- 172.21.15.23:0/4142245058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 msgr2=0x7f2178100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 --2- 172.21.15.23:0/4142245058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178100910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f21680099c0 tx=0x7f2168009cd0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 -- 172.21.15.23:0/4142245058 shutdown_connections 2023-11-17T00:00:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 --2- 172.21.15.23:0/4142245058 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21781055e0 0x7f217810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 --2- 172.21.15.23:0/4142245058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178100910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 --2- 172.21.15.23:0/4142245058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2178100ee0 0x7f2178104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 -- 172.21.15.23:0/4142245058 >> 172.21.15.23:0/4142245058 conn(0x7f21780fbdc0 msgr2=0x7f21780fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 -- 172.21.15.23:0/4142245058 shutdown_connections 2023-11-17T00:00:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.300+0000 7f217e953700 1 -- 172.21.15.23:0/4142245058 wait complete. 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.301+0000 7f217e953700 1 Processor -- start 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.301+0000 7f217e953700 1 -- start start 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 0x7f2178073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21781a47a0 con 0x7f2178100540 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21781a4910 con 0x7f2178100ee0 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217e953700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21781a4ad0 con 0x7f21781055e0 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f21777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f217cef0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 0x7f2178073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f2177fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f21777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:44396/0 (socket says 172.21.15.23:44396) 2023-11-17T00:00:32.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.302+0000 7f21777fe700 1 -- 172.21.15.23:0/2849268988 learned_addr learned my addr 172.21.15.23:0/2849268988 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:32.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 -- 172.21.15.23:0/2849268988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 msgr2=0x7f2178073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:32.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 0x7f2178073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 -- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 msgr2=0x7f2178072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 -- 172.21.15.23:0/2849268988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680095f0 con 0x7f2178100ee0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f2177fff700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f217cef0700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 0x7f2178073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21777fe700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f216c00d950 tx=0x7f216c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c004220 con 0x7f2178100ee0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2178110320 con 0x7f2178100ee0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f216c007620 con 0x7f2178100ee0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0056c0 con 0x7f2178100ee0 2023-11-17T00:00:32.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.303+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2178110190 con 0x7f2178100ee0 2023-11-17T00:00:32.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.304+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21781104b0 con 0x7f2178100ee0 2023-11-17T00:00:32.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.307+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f216c004380 con 0x7f2178100ee0 2023-11-17T00:00:32.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.308+0000 7f21757fa700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 0x7f21600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:32.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.308+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f216c0ccfa0 con 0x7f2178100ee0 2023-11-17T00:00:32.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.308+0000 7f2177fff700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 0x7f21600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:32.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.309+0000 7f2177fff700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 0x7f21600a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2168009990 tx=0x7f2168000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:32.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.310+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f216c096220 con 0x7f2178100ee0 2023-11-17T00:00:32.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-11-17T00:00:32.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: osdmap e89: 12 total, 12 up, 12 in 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-17T00:00:32.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:32 smithi173 ceph-mon[131534]: osdmap e90: 12 total, 12 up, 12 in 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-11-17T00:00:32.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: osdmap e89: 12 total, 12 up, 12 in 2023-11-17T00:00:32.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-17T00:00:32.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:32 smithi156 ceph-mon[131105]: osdmap e90: 12 total, 12 up, 12 in 2023-11-17T00:00:32.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.560+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2178066ea0 con 0x7f2178100ee0 2023-11-17T00:00:32.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.560+0000 7f21757fa700 1 -- 172.21.15.23:0/2849268988 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f216c016080 con 0x7f2178100ee0 2023-11-17T00:00:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:32.561 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 msgr2=0x7f21600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 0x7f21600a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2168009990 tx=0x7f2168000bc0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 msgr2=0x7f2178073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f216c00d950 tx=0x7f216c00dc60 comp rx=0 tx=0).stop 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 shutdown_connections 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f21600a1260 0x7f21600a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21781055e0 0x7f2178073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2178100540 0x7f2178072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 --2- 172.21.15.23:0/2849268988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2178100ee0 0x7f2178073330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 >> 172.21.15.23:0/2849268988 conn(0x7f21780fbdc0 msgr2=0x7f21780fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 shutdown_connections 2023-11-17T00:00:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:32.563+0000 7f217e953700 1 -- 172.21.15.23:0/2849268988 wait complete. 2023-11-17T00:00:33.141 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-17T00:00:33.142 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:00:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 -- 172.21.15.23:0/2441156139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1002b0 msgr2=0x7f8f1c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 --2- 172.21.15.23:0/2441156139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c1006c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009990 tx=0x7f8f0c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 -- 172.21.15.23:0/2441156139 shutdown_connections 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 --2- 172.21.15.23:0/2441156139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 --2- 172.21.15.23:0/2441156139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c1006c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 --2- 172.21.15.23:0/2441156139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.351+0000 7f8f21fff700 1 -- 172.21.15.23:0/2441156139 >> 172.21.15.23:0/2441156139 conn(0x7f8f1c0fb890 msgr2=0x7f8f1c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.352+0000 7f8f21fff700 1 -- 172.21.15.23:0/2441156139 shutdown_connections 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.352+0000 7f8f21fff700 1 -- 172.21.15.23:0/2441156139 wait complete. 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.353+0000 7f8f21fff700 1 Processor -- start 2023-11-17T00:00:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.353+0000 7f8f21fff700 1 -- start start 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c19dbc0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c19dd00 con 0x7f8f1c101e30 2023-11-17T00:00:33.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f21fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c19de90 con 0x7f8f1c1002b0 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54388/0 (socket says 172.21.15.23:54388) 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:51968/0 (socket says 172.21.15.23:51968) 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1affd700 1 -- 172.21.15.23:0/3587720093 learned_addr learned my addr 172.21.15.23:0/3587720093 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.354+0000 7f8f1bfff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 -- 172.21.15.23:0/3587720093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 msgr2=0x7f8f1c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 -- 172.21.15.23:0/3587720093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 msgr2=0x7f8f1c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 -- 172.21.15.23:0/3587720093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0095f0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1bfff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1b7fe700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:33.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f1affd700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000d950 tx=0x7f8f1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:33.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:33 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2849268988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T00:00:33.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:33 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2849268988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009940 con 0x7f8f1c1014a0 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f10010460 con 0x7f8f1c1014a0 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c10bc80 con 0x7f8f1c1014a0 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1000f620 con 0x7f8f1c1014a0 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.355+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c10baf0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.356+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c1078c0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.358+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8f100108d0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.360+0000 7f8f18ff9700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 0x7f8f040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.360+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8f100ccc10 con 0x7f8f1c1014a0 2023-11-17T00:00:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.360+0000 7f8f1b7fe700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 0x7f8f040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.361+0000 7f8f1b7fe700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 0x7f8f040a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00b5c0 tx=0x7f8f0c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.364+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8f10095e90 con 0x7f8f1c1014a0 2023-11-17T00:00:33.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:33 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2849268988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T00:00:33.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.573+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f1c02cfe0 con 0x7f8f1c1014a0 2023-11-17T00:00:33.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.575+0000 7f8f18ff9700 1 -- 172.21.15.23:0/3587720093 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7f8f10016090 con 0x7f8f1c1014a0 2023-11-17T00:00:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:33.576 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:00:33.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 msgr2=0x7f8f040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 0x7f8f040a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c00b5c0 tx=0x7f8f0c005fd0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 msgr2=0x7f8f1c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000d950 tx=0x7f8f1000dc60 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 shutdown_connections 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1c101e30 0x7f8f1c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8f040a1260 0x7f8f040a3710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f1c1002b0 0x7f8f1c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 --2- 172.21.15.23:0/3587720093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f1c1014a0 0x7f8f1c196580 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.577+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 >> 172.21.15.23:0/3587720093 conn(0x7f8f1c0fb890 msgr2=0x7f8f1c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.578+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 shutdown_connections 2023-11-17T00:00:33.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.578+0000 7f8f21fff700 1 -- 172.21.15.23:0/3587720093 wait complete. 2023-11-17T00:00:33.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:00:33.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:00:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 -- 172.21.15.23:0/2321564344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 msgr2=0x7f60001006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 --2- 172.21.15.23:0/2321564344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f60001006d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009990 tx=0x7f5ff0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 -- 172.21.15.23:0/2321564344 shutdown_connections 2023-11-17T00:00:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 --2- 172.21.15.23:0/2321564344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6000101e40 0x7f6000108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 --2- 172.21.15.23:0/2321564344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f60001006d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 --2- 172.21.15.23:0/2321564344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60001014b0 0x7f6000101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 -- 172.21.15.23:0/2321564344 >> 172.21.15.23:0/2321564344 conn(0x7f60000fb8a0 msgr2=0x7f60000fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 -- 172.21.15.23:0/2321564344 shutdown_connections 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.805+0000 7f6005fff700 1 -- 172.21.15.23:0/2321564344 wait complete. 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.806+0000 7f6005fff700 1 Processor -- start 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.806+0000 7f6005fff700 1 -- start start 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60001014b0 0x7f60001965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6000101e40 0x7f600019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600019dc10 con 0x7f60001002c0 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600019dd50 con 0x7f6000101e40 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f6005fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600019dee0 con 0x7f60001014b0 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54404/0 (socket says 172.21.15.23:54404) 2023-11-17T00:00:33.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 -- 172.21.15.23:0/1215790243 learned_addr learned my addr 172.21.15.23:0/1215790243 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5ff7fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60001014b0 0x7f60001965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5ffffff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6000101e40 0x7f600019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 -- 172.21.15.23:0/1215790243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60001014b0 msgr2=0x7f60001965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60001014b0 0x7f60001965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 -- 172.21.15.23:0/1215790243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6000101e40 msgr2=0x7f600019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6000101e40 0x7f600019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.807+0000 7f5fff7fe700 1 -- 172.21.15.23:0/1215790243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff00095f0 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f5fff7fe700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00098e0 tx=0x7f5ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff001d070 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ff0005030 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600010bc90 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0004b50 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.808+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600010bb00 con 0x7f60001002c0 2023-11-17T00:00:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.809+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6000107850 con 0x7f60001002c0 2023-11-17T00:00:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.811+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5ff0004d30 con 0x7f60001002c0 2023-11-17T00:00:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.812+0000 7f5ffd7fa700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 0x7f5fec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.813+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ff00ce070 con 0x7f60001002c0 2023-11-17T00:00:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.813+0000 7f5ff7fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 0x7f5fec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.813+0000 7f5ff7fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 0x7f5fec0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8005950 tx=0x7f5fe800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:33.817+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5ff005a730 con 0x7f60001002c0 2023-11-17T00:00:34.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.025+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6000061700 con 0x7f60001002c0 2023-11-17T00:00:34.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.025+0000 7f5ffd7fa700 1 -- 172.21.15.23:0/1215790243 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ff009ae10 con 0x7f60001002c0 2023-11-17T00:00:34.026 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:00:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 msgr2=0x7f5fec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 0x7f5fec0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8005950 tx=0x7f5fe800b410 comp rx=0 tx=0).stop 2023-11-17T00:00:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 msgr2=0x7f6000196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00098e0 tx=0x7f5ff0004730 comp rx=0 tx=0).stop 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 shutdown_connections 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6000101e40 0x7f600019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60001002c0 0x7f6000196090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f5fec0a11e0 0x7f5fec0a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 --2- 172.21.15.23:0/1215790243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60001014b0 0x7f60001965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 >> 172.21.15.23:0/1215790243 conn(0x7f60000fb8a0 msgr2=0x7f6000104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 shutdown_connections 2023-11-17T00:00:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.028+0000 7f6005fff700 1 -- 172.21.15.23:0/1215790243 wait complete. 2023-11-17T00:00:34.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:00:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 -- 172.21.15.23:0/2787477630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4658101e40 msgr2=0x7f4658108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 --2- 172.21.15.23:0/2787477630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4658101e40 0x7f4658108700 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f465000b0a0 tx=0x7f465000b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 -- 172.21.15.23:0/2787477630 shutdown_connections 2023-11-17T00:00:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 --2- 172.21.15.23:0/2787477630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4658101e40 0x7f4658108700 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 --2- 172.21.15.23:0/2787477630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46581002c0 0x7f46581006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 --2- 172.21.15.23:0/2787477630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 0x7f4658101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 -- 172.21.15.23:0/2787477630 >> 172.21.15.23:0/2787477630 conn(0x7f46580fb8a0 msgr2=0x7f46580fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.254+0000 7f465ffff700 1 -- 172.21.15.23:0/2787477630 shutdown_connections 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.255+0000 7f465ffff700 1 -- 172.21.15.23:0/2787477630 wait complete. 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.256+0000 7f465ffff700 1 Processor -- start 2023-11-17T00:00:34.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.256+0000 7f465ffff700 1 -- start start 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 0x7f4658196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4658101e40 0x7f465819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465819dcc0 con 0x7f46581002c0 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465819de00 con 0x7f4658101e40 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465ffff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465819df90 con 0x7f46581014b0 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54430/0 (socket says 172.21.15.23:54430) 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 -- 172.21.15.23:0/845073019 learned_addr learned my addr 172.21.15.23:0/845073019 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465d59a700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 0x7f4658196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465e59c700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4658101e40 0x7f465819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 -- 172.21.15.23:0/845073019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 msgr2=0x7f4658196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 0x7f4658196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 -- 172.21.15.23:0/845073019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4658101e40 msgr2=0x7f465819d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4658101e40 0x7f465819d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.257+0000 7f465dd9b700 1 -- 172.21.15.23:0/845073019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650009d00 con 0x7f46581002c0 2023-11-17T00:00:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f465dd9b700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f465400e970 tx=0x7f465400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654009790 con 0x7f46581002c0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f465400c7e0 con 0x7f46581002c0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654003c70 con 0x7f46581002c0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465810bc90 con 0x7f46581002c0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.258+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465810bb00 con 0x7f46581002c0 2023-11-17T00:00:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.259+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465819e690 con 0x7f46581002c0 2023-11-17T00:00:34.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.261+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f465400cc50 con 0x7f46581002c0 2023-11-17T00:00:34.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.262+0000 7f464effd700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 0x7f46440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.263+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4654014070 con 0x7f46581002c0 2023-11-17T00:00:34.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.263+0000 7f465d59a700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 0x7f46440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.263+0000 7f465d59a700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 0x7f46440a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4648009c60 tx=0x7f4648009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:34.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.266+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4654096220 con 0x7f46581002c0 2023-11-17T00:00:34.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:34 smithi173 ceph-mon[131534]: pgmap v351: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-11-17T00:00:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:34 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3587720093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:34.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:34 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1215790243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:34 smithi156 ceph-mon[131105]: pgmap v351: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-11-17T00:00:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:34 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3587720093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:34.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:34 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1215790243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:34.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.475+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46581070d0 con 0x7f46581002c0 2023-11-17T00:00:34.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.476+0000 7f464effd700 1 -- 172.21.15.23:0/845073019 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4654003dd0 con 0x7f46581002c0 2023-11-17T00:00:34.476 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:00:34.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 msgr2=0x7f46440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 0x7f46440a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4648009c60 tx=0x7f4648009260 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 msgr2=0x7f4658196140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f465400e970 tx=0x7f465400ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 shutdown_connections 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4658101e40 0x7f465819d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46581002c0 0x7f4658196140 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f46440a11e0 0x7f46440a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 --2- 172.21.15.23:0/845073019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46581014b0 0x7f4658196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 >> 172.21.15.23:0/845073019 conn(0x7f46580fb8a0 msgr2=0x7f46580fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 shutdown_connections 2023-11-17T00:00:34.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.478+0000 7f465ffff700 1 -- 172.21.15.23:0/845073019 wait complete. 2023-11-17T00:00:34.495 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:34 smithi023 ceph-mon[97824]: pgmap v351: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-11-17T00:00:34.495 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:34 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3587720093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:34.495 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:34 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1215790243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:34.499 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:00:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.705+0000 7f736b7ff700 1 -- 172.21.15.23:0/3321841703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 msgr2=0x7f7364108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.705+0000 7f736b7ff700 1 --2- 172.21.15.23:0/3321841703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f7364108700 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f735c00b0a0 tx=0x7f735c00b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.705+0000 7f736b7ff700 1 -- 172.21.15.23:0/3321841703 shutdown_connections 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 --2- 172.21.15.23:0/3321841703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f7364108700 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 --2- 172.21.15.23:0/3321841703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f73641006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 --2- 172.21.15.23:0/3321841703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 -- 172.21.15.23:0/3321841703 >> 172.21.15.23:0/3321841703 conn(0x7f73640fb8a0 msgr2=0x7f73640fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 -- 172.21.15.23:0/3321841703 shutdown_connections 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.706+0000 7f736b7ff700 1 -- 172.21.15.23:0/3321841703 wait complete. 2023-11-17T00:00:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.707+0000 7f736b7ff700 1 Processor -- start 2023-11-17T00:00:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.707+0000 7f736b7ff700 1 -- start start 2023-11-17T00:00:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f7364198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f736419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736419fec0 con 0x7f7364101e40 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73641a0000 con 0x7f73641014b0 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736b7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73641a0190 con 0x7f73641002c0 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736959b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f7364198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736959b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f7364198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52032/0 (socket says 172.21.15.23:52032) 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f736959b700 1 -- 172.21.15.23:0/2692496936 learned_addr learned my addr 172.21.15.23:0/2692496936 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f7369d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f736419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.708+0000 7f7368d9a700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 -- 172.21.15.23:0/2692496936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 msgr2=0x7f7364198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f7364198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 -- 172.21.15.23:0/2692496936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 msgr2=0x7f7364198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 -- 172.21.15.23:0/2692496936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c009d00 con 0x7f7364101e40 2023-11-17T00:00:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7368d9a700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f7369d9c700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f736419f8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f735c00b070 tx=0x7f735c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c00e040 con 0x7f7364101e40 2023-11-17T00:00:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f735c004e70 con 0x7f7364101e40 2023-11-17T00:00:34.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736410bc90 con 0x7f7364101e40 2023-11-17T00:00:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c004020 con 0x7f7364101e40 2023-11-17T00:00:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.709+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736410bb00 con 0x7f7364101e40 2023-11-17T00:00:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.710+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7364067b00 con 0x7f7364101e40 2023-11-17T00:00:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.712+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f735c01d690 con 0x7f7364101e40 2023-11-17T00:00:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.714+0000 7f735a7fc700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 0x7f73500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.714+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f735c0ced10 con 0x7f7364101e40 2023-11-17T00:00:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.714+0000 7f736959b700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 0x7f73500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.714+0000 7f736959b700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 0x7f73500a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7360007cf0 tx=0x7f7360007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:34.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.716+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f735c098020 con 0x7f7364101e40 2023-11-17T00:00:34.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.926+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73641a0ce0 con 0x7f7364101e40 2023-11-17T00:00:34.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.927+0000 7f735a7fc700 1 -- 172.21.15.23:0/2692496936 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f735c09b640 con 0x7f7364101e40 2023-11-17T00:00:34.927 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 msgr2=0x7f73500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 0x7f73500a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7360007cf0 tx=0x7f7360007480 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 msgr2=0x7f736419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f736419f8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f735c00b070 tx=0x7f735c004c80 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 shutdown_connections 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7364101e40 0x7f736419f8f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f73500a11e0 0x7f73500a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73641002c0 0x7f7364198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 --2- 172.21.15.23:0/2692496936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73641014b0 0x7f7364198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 >> 172.21.15.23:0/2692496936 conn(0x7f73640fb8a0 msgr2=0x7f7364106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:34.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 shutdown_connections 2023-11-17T00:00:34.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:34.929+0000 7f736b7ff700 1 -- 172.21.15.23:0/2692496936 wait complete. 2023-11-17T00:00:34.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 -- 172.21.15.23:0/3129836092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 msgr2=0x7f37841006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 --2- 172.21.15.23:0/3129836092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37841006d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f37740098f0 tx=0x7f3774009c00 comp rx=0 tx=0).stop 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 -- 172.21.15.23:0/3129836092 shutdown_connections 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 --2- 172.21.15.23:0/3129836092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3784101e40 0x7f3784108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 --2- 172.21.15.23:0/3129836092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37841006d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 --2- 172.21.15.23:0/3129836092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37841014b0 0x7f3784101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 -- 172.21.15.23:0/3129836092 >> 172.21.15.23:0/3129836092 conn(0x7f37840fb8a0 msgr2=0x7f37840fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:35.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 -- 172.21.15.23:0/3129836092 shutdown_connections 2023-11-17T00:00:35.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.157+0000 7f37893ff700 1 -- 172.21.15.23:0/3129836092 wait complete. 2023-11-17T00:00:35.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.158+0000 7f37893ff700 1 Processor -- start 2023-11-17T00:00:35.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.158+0000 7f37893ff700 1 -- start start 2023-11-17T00:00:35.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37841014b0 0x7f3784072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3784101e40 0x7f3784073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784073bc0 con 0x7f3784101e40 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784073d30 con 0x7f37841002c0 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37893ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841a0140 con 0x7f37841014b0 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f3782ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3784101e40 0x7f3784073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f3782ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:44460/0 (socket says 172.21.15.23:44460) 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f3782ffd700 1 -- 172.21.15.23:0/3486694194 learned_addr learned my addr 172.21.15.23:0/3486694194 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:35.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.159+0000 7f37827fc700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37841014b0 0x7f3784072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 -- 172.21.15.23:0/3486694194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37841014b0 msgr2=0x7f3784072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37841014b0 0x7f3784072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 -- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3784101e40 msgr2=0x7f3784073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3784101e40 0x7f3784073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 -- 172.21.15.23:0/3486694194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780095f0 con 0x7f37841002c0 2023-11-17T00:00:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f3782ffd700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3774005e90 tx=0x7f377400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377401c070 con 0x7f37841002c0 2023-11-17T00:00:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f377400f7f0 con 0x7f37841002c0 2023-11-17T00:00:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3774009520 con 0x7f37841002c0 2023-11-17T00:00:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774003990 con 0x7f37841002c0 2023-11-17T00:00:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.160+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378410bc90 con 0x7f37841002c0 2023-11-17T00:00:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.161+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784067b00 con 0x7f37841002c0 2023-11-17T00:00:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.163+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3774020030 con 0x7f37841002c0 2023-11-17T00:00:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.165+0000 7f376bfff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 0x7f376c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.165+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f37740ce620 con 0x7f37841002c0 2023-11-17T00:00:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.165+0000 7f37827fc700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 0x7f376c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.166+0000 7f37827fc700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 0x7f376c0a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3778005fd0 tx=0x7f3778009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.167+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3774097930 con 0x7f37841002c0 2023-11-17T00:00:35.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:35 smithi023 ceph-mon[97824]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2023-11-17T00:00:35.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:35 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/845073019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:35 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2692496936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:35 smithi173 ceph-mon[131534]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2023-11-17T00:00:35.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:35 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/845073019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:35 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2692496936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:35 smithi156 ceph-mon[131105]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2023-11-17T00:00:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:35 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/845073019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:35 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2692496936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.379+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f37841a0cc0 con 0x7f37841002c0 2023-11-17T00:00:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.380+0000 7f376bfff700 1 -- 172.21.15.23:0/3486694194 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f377409af50 con 0x7f37841002c0 2023-11-17T00:00:35.381 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:00:35.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 msgr2=0x7f376c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 0x7f376c0a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3778005fd0 tx=0x7f3778009310 comp rx=0 tx=0).stop 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 msgr2=0x7f37840748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3774005e90 tx=0x7f377400b730 comp rx=0 tx=0).stop 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 shutdown_connections 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3784101e40 0x7f3784073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37841002c0 0x7f37840748a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f376c0a1260 0x7f376c0a3710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 --2- 172.21.15.23:0/3486694194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37841014b0 0x7f3784072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.382+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 >> 172.21.15.23:0/3486694194 conn(0x7f37840fb8a0 msgr2=0x7f37840fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.383+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 shutdown_connections 2023-11-17T00:00:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.383+0000 7f37893ff700 1 -- 172.21.15.23:0/3486694194 wait complete. 2023-11-17T00:00:35.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:00:35.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 -- 172.21.15.23:0/1075075644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 msgr2=0x7f3ab81018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/1075075644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81018f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009990 tx=0x7f3aa8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 -- 172.21.15.23:0/1075075644 shutdown_connections 2023-11-17T00:00:35.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/1075075644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101e30 0x7f3ab81086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/1075075644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab81002b0 0x7f3ab81006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/1075075644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81018f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 -- 172.21.15.23:0/1075075644 >> 172.21.15.23:0/1075075644 conn(0x7f3ab80fb890 msgr2=0x7f3ab80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:35.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 -- 172.21.15.23:0/1075075644 shutdown_connections 2023-11-17T00:00:35.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.607+0000 7f3ac05ff700 1 -- 172.21.15.23:0/1075075644 wait complete. 2023-11-17T00:00:35.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 Processor -- start 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 -- start start 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab8101e30 0x7f3ab819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab819fd30 con 0x7f3ab81014a0 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab819fe70 con 0x7f3ab8101e30 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.608+0000 7f3ac05ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81a0000 con 0x7f3ab81002b0 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abdb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52070/0 (socket says 172.21.15.23:52070) 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 -- 172.21.15.23:0/2221728625 learned_addr learned my addr 172.21.15.23:0/2221728625 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abeb9c700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab8101e30 0x7f3ab819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 -- 172.21.15.23:0/2221728625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab8101e30 msgr2=0x7f3ab819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab8101e30 0x7f3ab819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 -- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 msgr2=0x7f3ab81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 -- 172.21.15.23:0/2221728625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80095f0 con 0x7f3ab81002b0 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.609+0000 7f3abe39b700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e970 tx=0x7f3ab400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.610+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4009790 con 0x7f3ab81002b0 2023-11-17T00:00:35.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.610+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab810bc60 con 0x7f3ab81002b0 2023-11-17T00:00:35.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.610+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ab400c7e0 con 0x7f3ab81002b0 2023-11-17T00:00:35.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.610+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4003c70 con 0x7f3ab81002b0 2023-11-17T00:00:35.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.610+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab810bad0 con 0x7f3ab81002b0 2023-11-17T00:00:35.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.611+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab81a0750 con 0x7f3ab81002b0 2023-11-17T00:00:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.613+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3ab400cc50 con 0x7f3ab81002b0 2023-11-17T00:00:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.614+0000 7f3aaf7fe700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 0x7f3aa40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.614+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3ab4014070 con 0x7f3ab81002b0 2023-11-17T00:00:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.614+0000 7f3abdb9a700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 0x7f3aa40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.615+0000 7f3abdb9a700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 0x7f3aa40a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:35.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.616+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3ab4096380 con 0x7f3ab81002b0 2023-11-17T00:00:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.869+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ab81a0d00 con 0x7f3ab81002b0 2023-11-17T00:00:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.871+0000 7f3aaf7fe700 1 -- 172.21.15.23:0/2221728625 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7f3ab4003dd0 con 0x7f3ab81002b0 2023-11-17T00:00:35.871 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:35.872 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:00:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.872+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 msgr2=0x7f3aa40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.872+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 0x7f3aa40a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa8005fb0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.872+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 msgr2=0x7f3ab81981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.872+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e970 tx=0x7f3ab400ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 shutdown_connections 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ab8101e30 0x7f3ab819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3aa40a11e0 0x7f3aa40a3690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab81002b0 0x7f3ab81981b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 --2- 172.21.15.23:0/2221728625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ab81014a0 0x7f3ab81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 >> 172.21.15.23:0/2221728625 conn(0x7f3ab80fb890 msgr2=0x7f3ab80fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 shutdown_connections 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:35.873+0000 7f3ac05ff700 1 -- 172.21.15.23:0/2221728625 wait complete. 2023-11-17T00:00:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:00:35.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:00:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 -- 172.21.15.23:0/3168683422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4100290 msgr2=0x7fcba41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3168683422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4100290 0x7fcba41006a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009990 tx=0x7fcb94009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 -- 172.21.15.23:0/3168683422 shutdown_connections 2023-11-17T00:00:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3168683422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4101e10 0x7fcba41086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3168683422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4100290 0x7fcba41006a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3168683422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 0x7fcba41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 -- 172.21.15.23:0/3168683422 >> 172.21.15.23:0/3168683422 conn(0x7fcba40fb870 msgr2=0x7fcba40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 -- 172.21.15.23:0/3168683422 shutdown_connections 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.100+0000 7fcba8bff700 1 -- 172.21.15.23:0/3168683422 wait complete. 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 Processor -- start 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 -- start start 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4100290 0x7fcba4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 0x7fcba4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba419fd80 con 0x7fcba4101e10 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.101+0000 7fcba8bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba419fec0 con 0x7fcba4100290 2023-11-17T00:00:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba8bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41a0050 con 0x7fcba4101480 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54512/0 (socket says 172.21.15.23:54512) 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 -- 172.21.15.23:0/3228011083 learned_addr learned my addr 172.21.15.23:0/3228011083 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba259c700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4100290 0x7fcba4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba1d9b700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 0x7fcba4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 -- 172.21.15.23:0/3228011083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 msgr2=0x7fcba4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 0x7fcba4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 -- 172.21.15.23:0/3228011083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4100290 msgr2=0x7fcba4198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4100290 0x7fcba4198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 -- 172.21.15.23:0/3228011083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940095f0 con 0x7fcba4101e10 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.102+0000 7fcba2d9d700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007ee0 tx=0x7fcb9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.103+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c017070 con 0x7fcba4101e10 2023-11-17T00:00:36.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.103+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb9c00f040 con 0x7fcba4101e10 2023-11-17T00:00:36.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.103+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba410bc40 con 0x7fcba4101e10 2023-11-17T00:00:36.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.103+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c005510 con 0x7fcba4101e10 2023-11-17T00:00:36.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.103+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba410bab0 con 0x7fcba4101e10 2023-11-17T00:00:36.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.104+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba41928d0 con 0x7fcba4101e10 2023-11-17T00:00:36.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.106+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcb9c00db20 con 0x7fcba4101e10 2023-11-17T00:00:36.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.107+0000 7fcb937fe700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 0x7fcb8c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.107+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb9c0cc4a0 con 0x7fcba4101e10 2023-11-17T00:00:36.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.108+0000 7fcba259c700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 0x7fcb8c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.108+0000 7fcba259c700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 0x7fcb8c0a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400b5c0 tx=0x7fcb94005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.111+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcb9c095720 con 0x7fcba4101e10 2023-11-17T00:00:36.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:36 smithi023 ceph-mon[97824]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-11-17T00:00:36.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:36 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3486694194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:36.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:36 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2221728625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:36.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.321+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcba402cfe0 con 0x7fcba4101e10 2023-11-17T00:00:36.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.323+0000 7fcb937fe700 1 -- 172.21.15.23:0/3228011083 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7fcb9c098d40 con 0x7fcba4101e10 2023-11-17T00:00:36.324 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:36.324 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 msgr2=0x7fcb8c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 0x7fcb8c0a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400b5c0 tx=0x7fcb94005fd0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 msgr2=0x7fcba419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007ee0 tx=0x7fcb9c00d3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 shutdown_connections 2023-11-17T00:00:36.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4101e10 0x7fcba419f7b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcba4100290 0x7fcba4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fcb8c0a11e0 0x7fcb8c0a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 --2- 172.21.15.23:0/3228011083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4101480 0x7fcba4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 >> 172.21.15.23:0/3228011083 conn(0x7fcba40fb870 msgr2=0x7fcba4106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 shutdown_connections 2023-11-17T00:00:36.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.325+0000 7fcba8bff700 1 -- 172.21.15.23:0/3228011083 wait complete. 2023-11-17T00:00:36.345 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:00:36.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:00:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:36 smithi173 ceph-mon[131534]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-11-17T00:00:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:36 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3486694194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:36 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2221728625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:36 smithi156 ceph-mon[131105]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-11-17T00:00:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:36 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3486694194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:36.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:36 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2221728625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:36.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.554+0000 7f722afff700 1 -- 172.21.15.23:0/399825209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241002c0 msgr2=0x7f72241006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.554+0000 7f722afff700 1 --2- 172.21.15.23:0/399825209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241002c0 0x7f72241006d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7218009990 tx=0x7f7218009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 -- 172.21.15.23:0/399825209 shutdown_connections 2023-11-17T00:00:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 --2- 172.21.15.23:0/399825209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 0x7f7224108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 --2- 172.21.15.23:0/399825209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241002c0 0x7f72241006d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 --2- 172.21.15.23:0/399825209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241014b0 0x7f7224101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 -- 172.21.15.23:0/399825209 >> 172.21.15.23:0/399825209 conn(0x7f72240fb8a0 msgr2=0x7f72240fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 -- 172.21.15.23:0/399825209 shutdown_connections 2023-11-17T00:00:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.555+0000 7f722afff700 1 -- 172.21.15.23:0/399825209 wait complete. 2023-11-17T00:00:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 Processor -- start 2023-11-17T00:00:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 -- start start 2023-11-17T00:00:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241002c0 0x7f7224198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 0x7f722419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722419fd90 con 0x7f7224101e40 2023-11-17T00:00:36.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722419fed0 con 0x7f72241002c0 2023-11-17T00:00:36.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722afff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72241a0060 con 0x7f72241014b0 2023-11-17T00:00:36.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f7223fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f722959c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 0x7f722419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f7223fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52108/0 (socket says 172.21.15.23:52108) 2023-11-17T00:00:36.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.556+0000 7f7223fff700 1 -- 172.21.15.23:0/2292854559 learned_addr learned my addr 172.21.15.23:0/2292854559 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:36.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7228d9b700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241002c0 0x7f7224198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 -- 172.21.15.23:0/2292854559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241002c0 msgr2=0x7f7224198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241002c0 0x7f7224198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 -- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 msgr2=0x7f722419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 0x7f722419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 -- 172.21.15.23:0/2292854559 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180095f0 con 0x7f72241014b0 2023-11-17T00:00:36.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7223fff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f721000e9a0 tx=0x7f721000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:36.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210009790 con 0x7f72241014b0 2023-11-17T00:00:36.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7210010430 con 0x7f72241014b0 2023-11-17T00:00:36.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722410bc10 con 0x7f72241014b0 2023-11-17T00:00:36.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.557+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210018680 con 0x7f72241014b0 2023-11-17T00:00:36.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.558+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722410ba80 con 0x7f72241014b0 2023-11-17T00:00:36.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.558+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72241a0750 con 0x7f72241014b0 2023-11-17T00:00:36.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.561+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f72100105a0 con 0x7f72241014b0 2023-11-17T00:00:36.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.562+0000 7f7221ffb700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 0x7f72140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:36.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.562+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7210014070 con 0x7f72241014b0 2023-11-17T00:00:36.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.562+0000 7f7228d9b700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 0x7f72140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:36.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.563+0000 7f7228d9b700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 0x7f72140a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f721800b2e0 tx=0x7f7218005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:36.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.564+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7210096440 con 0x7f72241014b0 2023-11-17T00:00:36.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.776+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72241a0d00 con 0x7f72241014b0 2023-11-17T00:00:36.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.777+0000 7f7221ffb700 1 -- 172.21.15.23:0/2292854559 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72100260a0 con 0x7f72241014b0 2023-11-17T00:00:36.778 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 msgr2=0x7f72140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 0x7f72140a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f721800b2e0 tx=0x7f7218005fb0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 msgr2=0x7f7224198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f721000e9a0 tx=0x7f721000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 shutdown_connections 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7224101e40 0x7f722419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72241002c0 0x7f7224198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f72140a11e0 0x7f72140a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 --2- 172.21.15.23:0/2292854559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72241014b0 0x7f7224198750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 >> 172.21.15.23:0/2292854559 conn(0x7f72240fb8a0 msgr2=0x7f72240fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 shutdown_connections 2023-11-17T00:00:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:36.780+0000 7f722afff700 1 -- 172.21.15.23:0/2292854559 wait complete. 2023-11-17T00:00:36.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:00:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 -- 172.21.15.23:0/3444115739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f810c8a0 msgr2=0x7f80f810ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 --2- 172.21.15.23:0/3444115739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f810c8a0 0x7f80f810ec80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b0a0 tx=0x7f80f000b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 -- 172.21.15.23:0/3444115739 shutdown_connections 2023-11-17T00:00:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 --2- 172.21.15.23:0/3444115739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f810c8a0 0x7f80f810ec80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 --2- 172.21.15.23:0/3444115739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8074b30 0x7f80f8072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 --2- 172.21.15.23:0/3444115739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f8073550 0x7f80f80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 -- 172.21.15.23:0/3444115739 >> 172.21.15.23:0/3444115739 conn(0x7f80f80fb8b0 msgr2=0x7f80f80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 -- 172.21.15.23:0/3444115739 shutdown_connections 2023-11-17T00:00:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.006+0000 7f80fdfff700 1 -- 172.21.15.23:0/3444115739 wait complete. 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.007+0000 7f80fdfff700 1 Processor -- start 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 -- start start 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8073550 0x7f80f819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f810c8a0 0x7f80f81a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f81a4300 con 0x7f80f8074b30 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f81a4440 con 0x7f80f810c8a0 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80fdfff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f81a45d0 con 0x7f80f8073550 2023-11-17T00:00:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54558/0 (socket says 172.21.15.23:54558) 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80f6ffd700 1 -- 172.21.15.23:0/2909502580 learned_addr learned my addr 172.21.15.23:0/2909502580 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80f77fe700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8073550 0x7f80f819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.008+0000 7f80f7fff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f810c8a0 0x7f80f81a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 -- 172.21.15.23:0/2909502580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8073550 msgr2=0x7f80f819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8073550 0x7f80f819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 -- 172.21.15.23:0/2909502580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f810c8a0 msgr2=0x7f80f81a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f810c8a0 0x7f80f81a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 -- 172.21.15.23:0/2909502580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0009d00 con 0x7f80f8074b30 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f6ffd700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e970 tx=0x7f80ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec009790 con 0x7f80f8074b30 2023-11-17T00:00:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80ec00c7e0 con 0x7f80f8074b30 2023-11-17T00:00:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f81122b0 con 0x7f80f8074b30 2023-11-17T00:00:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec003c70 con 0x7f80f8074b30 2023-11-17T00:00:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.009+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f8112120 con 0x7f80f8074b30 2023-11-17T00:00:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.010+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f81023f0 con 0x7f80f8074b30 2023-11-17T00:00:37.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.012+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f80ec00cc50 con 0x7f80f8074b30 2023-11-17T00:00:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.014+0000 7f80f4ff9700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 0x7f80e00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.014+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f80ec014070 con 0x7f80f8074b30 2023-11-17T00:00:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.014+0000 7f80f77fe700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 0x7f80e00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.015+0000 7f80f77fe700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 0x7f80e00a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f80f8074f00 tx=0x7f80e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.017+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f80ec0961a0 con 0x7f80f8074b30 2023-11-17T00:00:37.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.229+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80f8061700 con 0x7f80f8074b30 2023-11-17T00:00:37.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.230+0000 7f80f4ff9700 1 -- 172.21.15.23:0/2909502580 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80ec003dd0 con 0x7f80f8074b30 2023-11-17T00:00:37.231 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:00:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.232+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 msgr2=0x7f80e00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.232+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 0x7f80e00a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f80f8074f00 tx=0x7f80e8009260 comp rx=0 tx=0).stop 2023-11-17T00:00:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.232+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 msgr2=0x7f80f819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.232+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e970 tx=0x7f80ec00ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 shutdown_connections 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f810c8a0 0x7f80f81a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f80e00a1260 0x7f80e00a3710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80f8073550 0x7f80f819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 --2- 172.21.15.23:0/2909502580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80f8074b30 0x7f80f819ccc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 >> 172.21.15.23:0/2909502580 conn(0x7f80f80fb8b0 msgr2=0x7f80f80fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 shutdown_connections 2023-11-17T00:00:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.233+0000 7f80fdfff700 1 -- 172.21.15.23:0/2909502580 wait complete. 2023-11-17T00:00:37.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:00:37.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:37 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3228011083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:37.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:37 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2292854559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:37.362 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:37 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3228011083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:37.362 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:37 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2292854559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 -- 172.21.15.23:0/430032466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 msgr2=0x7fc3a4101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/430032466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a4101910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009990 tx=0x7fc38c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 -- 172.21.15.23:0/430032466 shutdown_connections 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/430032466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a4101e50 0x7fc3a4108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/430032466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a41002d0 0x7fc3a41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/430032466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a4101910 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 -- 172.21.15.23:0/430032466 >> 172.21.15.23:0/430032466 conn(0x7fc3a40fb8b0 msgr2=0x7fc3a40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 -- 172.21.15.23:0/430032466 shutdown_connections 2023-11-17T00:00:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.458+0000 7fc3a8dff700 1 -- 172.21.15.23:0/430032466 wait complete. 2023-11-17T00:00:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.459+0000 7fc3a8dff700 1 Processor -- start 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.459+0000 7fc3a8dff700 1 -- start start 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 0x7fc3a4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4101e50 0x7fc3a419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a419fe00 con 0x7fc3a41014c0 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a419ff40 con 0x7fc3a41002d0 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a8dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41a00d0 con 0x7fc3a4101e50 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54578/0 (socket says 172.21.15.23:54578) 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 -- 172.21.15.23:0/3253532283 learned_addr learned my addr 172.21.15.23:0/3253532283 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a2d9d700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4101e50 0x7fc3a419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a259c700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 0x7fc3a4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 -- 172.21.15.23:0/3253532283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4101e50 msgr2=0x7fc3a419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4101e50 0x7fc3a419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 -- 172.21.15.23:0/3253532283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 msgr2=0x7fc3a4198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 0x7fc3a4198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc39bfff700 1 -- 172.21.15.23:0/3253532283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0095f0 con 0x7fc3a41014c0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.460+0000 7fc3a259c700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 0x7fc3a4198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc39bfff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00c010 tx=0x7fc38c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c01d070 con 0x7fc3a41014c0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc38c0054f0 con 0x7fc3a41014c0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c004ca0 con 0x7fc3a41014c0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a410bc80 con 0x7fc3a41014c0 2023-11-17T00:00:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.461+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a410baf0 con 0x7fc3a41014c0 2023-11-17T00:00:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.462+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a41a06d0 con 0x7fc3a41014c0 2023-11-17T00:00:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.464+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc38c00fbc0 con 0x7fc3a41014c0 2023-11-17T00:00:37.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.465+0000 7fc39b7fe700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 0x7fc3900a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.466+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc38c0ce490 con 0x7fc3a41014c0 2023-11-17T00:00:37.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.466+0000 7fc3a259c700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 0x7fc3900a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.466+0000 7fc3a259c700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 0x7fc3900a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4101320 tx=0x7fc394009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.467+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc38c097800 con 0x7fc3a41014c0 2023-11-17T00:00:37.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:37 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3228011083' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:37 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2292854559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:00:37.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.678+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3a41a0c80 con 0x7fc3a41014c0 2023-11-17T00:00:37.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.678+0000 7fc39b7fe700 1 -- 172.21.15.23:0/3253532283 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc38c09ae20 con 0x7fc3a41014c0 2023-11-17T00:00:37.679 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:00:37.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 msgr2=0x7fc3900a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 0x7fc3900a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4101320 tx=0x7fc394009260 comp rx=0 tx=0).stop 2023-11-17T00:00:37.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 msgr2=0x7fc3a41987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00c010 tx=0x7fc38c004730 comp rx=0 tx=0).stop 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 shutdown_connections 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc3900a1260 0x7fc3900a3710 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4101e50 0x7fc3a419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3a41002d0 0x7fc3a4198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.680+0000 7fc3a8dff700 1 --2- 172.21.15.23:0/3253532283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3a41014c0 0x7fc3a41987c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.681+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 >> 172.21.15.23:0/3253532283 conn(0x7fc3a40fb8b0 msgr2=0x7fc3a40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:37.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.681+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 shutdown_connections 2023-11-17T00:00:37.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.681+0000 7fc3a8dff700 1 -- 172.21.15.23:0/3253532283 wait complete. 2023-11-17T00:00:37.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 -- 172.21.15.23:0/3114754813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c074b30 msgr2=0x7f0f1c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3114754813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c074b30 0x7f0f1c072f80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009990 tx=0x7f0f18009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 -- 172.21.15.23:0/3114754813 shutdown_connections 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3114754813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3114754813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c074b30 0x7f0f1c072f80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3114754813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c073550 0x7f0f1c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 -- 172.21.15.23:0/3114754813 >> 172.21.15.23:0/3114754813 conn(0x7f0f1c0fb8a0 msgr2=0x7f0f1c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 -- 172.21.15.23:0/3114754813 shutdown_connections 2023-11-17T00:00:37.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.910+0000 7f0f23bff700 1 -- 172.21.15.23:0/3114754813 wait complete. 2023-11-17T00:00:37.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 Processor -- start 2023-11-17T00:00:37.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 -- start start 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c074b30 0x7f0f1c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c19fd60 con 0x7f0f1c073550 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c19fea0 con 0x7f0f1c074b30 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.912+0000 7f0f23bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1a0030 con 0x7f0f1c108180 2023-11-17T00:00:37.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54604/0 (socket says 172.21.15.23:54604) 2023-11-17T00:00:37.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 -- 172.21.15.23:0/3797995568 learned_addr learned my addr 172.21.15.23:0/3797995568 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:37.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2219c700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2119a700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c074b30 0x7f0f1c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 -- 172.21.15.23:0/3797995568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 msgr2=0x7f0f1c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 -- 172.21.15.23:0/3797995568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c074b30 msgr2=0x7f0f1c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c074b30 0x7f0f1c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 -- 172.21.15.23:0/3797995568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180095f0 con 0x7f0f1c073550 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2219c700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.913+0000 7f0f2199b700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f18000c00 tx=0x7f0f18005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.914+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1801d070 con 0x7f0f1c073550 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.914+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f18022470 con 0x7f0f1c073550 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.914+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c10de30 con 0x7f0f1c073550 2023-11-17T00:00:37.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.914+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18017650 con 0x7f0f1c073550 2023-11-17T00:00:37.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.914+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c10dca0 con 0x7f0f1c073550 2023-11-17T00:00:37.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.915+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c106830 con 0x7f0f1c073550 2023-11-17T00:00:37.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.917+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0f180225e0 con 0x7f0f1c073550 2023-11-17T00:00:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.918+0000 7f0f12ffd700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 0x7f0f080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.918+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f180ce170 con 0x7f0f1c073550 2023-11-17T00:00:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.918+0000 7f0f2119a700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 0x7f0f080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.919+0000 7f0f2119a700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 0x7f0f080a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c074850 tx=0x7f0f0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:37.922+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0f1805c730 con 0x7f0f1c073550 2023-11-17T00:00:38.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.133+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f1c061700 con 0x7f0f1c073550 2023-11-17T00:00:38.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.133+0000 7f0f12ffd700 1 -- 172.21.15.23:0/3797995568 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f18097480 con 0x7f0f1c073550 2023-11-17T00:00:38.134 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 msgr2=0x7f0f080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 0x7f0f080a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c074850 tx=0x7f0f0c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 msgr2=0x7f0f1c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f18000c00 tx=0x7f0f18005720 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 shutdown_connections 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f080a11e0 0x7f0f080a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f1c108180 0x7f0f1c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1c073550 0x7f0f1c1981e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 --2- 172.21.15.23:0/3797995568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f1c074b30 0x7f0f1c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 >> 172.21.15.23:0/3797995568 conn(0x7f0f1c0fb8a0 msgr2=0x7f0f1c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 shutdown_connections 2023-11-17T00:00:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.136+0000 7f0f23bff700 1 -- 172.21.15.23:0/3797995568 wait complete. 2023-11-17T00:00:38.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:00:38.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:38 smithi173 ceph-mon[131534]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:38.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:38 smithi173 ceph-mon[131534]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2023-11-17T00:00:38.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:38 smithi173 ceph-mon[131534]: Cluster is now healthy 2023-11-17T00:00:38.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:38 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2909502580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:38 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3253532283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:38 smithi156 ceph-mon[131105]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:38.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:38 smithi156 ceph-mon[131105]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2023-11-17T00:00:38.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:38 smithi156 ceph-mon[131105]: Cluster is now healthy 2023-11-17T00:00:38.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:38 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2909502580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.361 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:38 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3253532283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 -- 172.21.15.23:0/3828952289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 msgr2=0x7fdeec0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 --2- 172.21.15.23:0/3828952289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec0739c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009990 tx=0x7fdee0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 -- 172.21.15.23:0/3828952289 shutdown_connections 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 --2- 172.21.15.23:0/3828952289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec10c860 0x7fdeec10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 --2- 172.21.15.23:0/3828952289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec074b30 0x7fdeec072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 --2- 172.21.15.23:0/3828952289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec0739c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 -- 172.21.15.23:0/3828952289 >> 172.21.15.23:0/3828952289 conn(0x7fdeec0fb870 msgr2=0x7fdeec0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.366+0000 7fdef0bff700 1 -- 172.21.15.23:0/3828952289 shutdown_connections 2023-11-17T00:00:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.367+0000 7fdef0bff700 1 -- 172.21.15.23:0/3828952289 wait complete. 2023-11-17T00:00:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 Processor -- start 2023-11-17T00:00:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 -- start start 2023-11-17T00:00:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec10c860 0x7fdeec1a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec1a4120 con 0x7fdeec074b30 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec1a4260 con 0x7fdeec073550 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdef0bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec1a43f0 con 0x7fdeec10c860 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdee9d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdee9d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54630/0 (socket says 172.21.15.23:54630) 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdeea59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.368+0000 7fdee9d9b700 1 -- 172.21.15.23:0/1572900817 learned_addr learned my addr 172.21.15.23:0/1572900817 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdeead9d700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec10c860 0x7fdeec1a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 -- 172.21.15.23:0/1572900817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec10c860 msgr2=0x7fdeec1a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec10c860 0x7fdeec1a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 -- 172.21.15.23:0/1572900817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 msgr2=0x7fdeec19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 -- 172.21.15.23:0/1572900817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00095f0 con 0x7fdeec074b30 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdeea59c700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdee9d9b700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdee0000c00 tx=0x7fdee0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee001d070 con 0x7fdeec074b30 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdee0022470 con 0x7fdeec074b30 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec112220 con 0x7fdeec074b30 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.369+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0017650 con 0x7fdeec074b30 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.370+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec112090 con 0x7fdeec074b30 2023-11-17T00:00:38.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.371+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec1023b0 con 0x7fdeec074b30 2023-11-17T00:00:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.373+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdee00225e0 con 0x7fdeec074b30 2023-11-17T00:00:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.374+0000 7fdedb7fe700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 0x7fded40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.374+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdee00cdfd0 con 0x7fdeec074b30 2023-11-17T00:00:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.374+0000 7fdeea59c700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 0x7fded40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.375+0000 7fdeea59c700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 0x7fded40a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdeec074f00 tx=0x7fdedc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.378+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdee005c730 con 0x7fdeec074b30 2023-11-17T00:00:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:38 smithi023 ceph-mon[97824]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:38 smithi023 ceph-mon[97824]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2023-11-17T00:00:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:38 smithi023 ceph-mon[97824]: Cluster is now healthy 2023-11-17T00:00:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:38 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2909502580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:38 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3253532283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:00:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.633+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeec02cfe0 con 0x7fdeec074b30 2023-11-17T00:00:38.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.634+0000 7fdedb7fe700 1 -- 172.21.15.23:0/1572900817 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7fdee0097340 con 0x7fdeec074b30 2023-11-17T00:00:38.650 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:38.651 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.636+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 msgr2=0x7fded40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.636+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 0x7fded40a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdeec074f00 tx=0x7fdedc009260 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.636+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 msgr2=0x7fdeec19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.636+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdee0000c00 tx=0x7fdee0005720 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 shutdown_connections 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fded40a1260 0x7fded40a3710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdeec10c860 0x7fdeec1a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdeec073550 0x7fdeec19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 --2- 172.21.15.23:0/1572900817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdeec074b30 0x7fdeec19cae0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 >> 172.21.15.23:0/1572900817 conn(0x7fdeec0fb870 msgr2=0x7fdeec0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 shutdown_connections 2023-11-17T00:00:38.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.637+0000 7fdef0bff700 1 -- 172.21.15.23:0/1572900817 wait complete. 2023-11-17T00:00:38.653 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:00:38.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:00:38.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 -- 172.21.15.23:0/2613020384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 msgr2=0x7fc0481018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/2613020384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc0481018d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009990 tx=0x7fc03c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 -- 172.21.15.23:0/2613020384 shutdown_connections 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/2613020384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048101e10 0x7fc0481086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/2613020384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048100290 0x7fc0481006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/2613020384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc0481018d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 -- 172.21.15.23:0/2613020384 >> 172.21.15.23:0/2613020384 conn(0x7fc0480fb870 msgr2=0x7fc0480fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 -- 172.21.15.23:0/2613020384 shutdown_connections 2023-11-17T00:00:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.863+0000 7fc04c9ff700 1 -- 172.21.15.23:0/2613020384 wait complete. 2023-11-17T00:00:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 Processor -- start 2023-11-17T00:00:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 -- start start 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 0x7fc0481981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 0x7fc04819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04819fd50 con 0x7fc048101480 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04819fe90 con 0x7fc048101e10 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.864+0000 7fc04c9ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0481a0020 con 0x7fc048100290 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc045d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 0x7fc0481981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc045d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54650/0 (socket says 172.21.15.23:54650) 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc045d9b700 1 -- 172.21.15.23:0/1333324320 learned_addr learned my addr 172.21.15.23:0/1333324320 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc046d9d700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 0x7fc04819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 -- 172.21.15.23:0/1333324320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 msgr2=0x7fc04819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 0x7fc04819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 -- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 msgr2=0x7fc048198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 -- 172.21.15.23:0/1333324320 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0095f0 con 0x7fc048100290 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc045d9b700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc046d9d700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 0x7fc04819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.865+0000 7fc04659c700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 0x7fc0481981d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc03800e9c0 tx=0x7fc03800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.866+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03800cbd0 con 0x7fc048100290 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.866+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04810bc50 con 0x7fc048100290 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.866+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc03800eeb0 con 0x7fc048100290 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.866+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038003c70 con 0x7fc048100290 2023-11-17T00:00:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.866+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04810bac0 con 0x7fc048100290 2023-11-17T00:00:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.867+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0481928e0 con 0x7fc048100290 2023-11-17T00:00:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.869+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc038003dd0 con 0x7fc048100290 2023-11-17T00:00:38.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.870+0000 7fc0377fe700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 0x7fc0300a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:38.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.871+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc038014070 con 0x7fc048100290 2023-11-17T00:00:38.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.871+0000 7fc045d9b700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 0x7fc0300a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:38.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.871+0000 7fc045d9b700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 0x7fc0300a3710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc03c006010 tx=0x7fc03c00b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:38.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:38.872+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc038096420 con 0x7fc048100290 2023-11-17T00:00:39.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.083+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc04802cfe0 con 0x7fc048100290 2023-11-17T00:00:39.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.085+0000 7fc0377fe700 1 -- 172.21.15.23:0/1333324320 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7fc038099a40 con 0x7fc048100290 2023-11-17T00:00:39.087 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:39.088 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 msgr2=0x7fc0300a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 0x7fc0300a3710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc03c006010 tx=0x7fc03c00b260 comp rx=0 tx=0).stop 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 msgr2=0x7fc0481981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 0x7fc0481981d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc03800e9c0 tx=0x7fc03800ecd0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 shutdown_connections 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc048101e10 0x7fc04819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc0300a1260 0x7fc0300a3710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc048100290 0x7fc0481981d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 --2- 172.21.15.23:0/1333324320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc048101480 0x7fc048198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 >> 172.21.15.23:0/1333324320 conn(0x7fc0480fb870 msgr2=0x7fc048106dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 shutdown_connections 2023-11-17T00:00:39.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.088+0000 7fc04c9ff700 1 -- 172.21.15.23:0/1333324320 wait complete. 2023-11-17T00:00:39.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:00:39.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.314+0000 7f40885ff700 1 -- 172.21.15.23:0/1921537518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 msgr2=0x7f40801086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.314+0000 7f40885ff700 1 --2- 172.21.15.23:0/1921537518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f40801086d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f407800b0a0 tx=0x7f407800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 -- 172.21.15.23:0/1921537518 shutdown_connections 2023-11-17T00:00:39.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 --2- 172.21.15.23:0/1921537518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f40801086d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 --2- 172.21.15.23:0/1921537518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 0x7f40801006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 --2- 172.21.15.23:0/1921537518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f40801018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 -- 172.21.15.23:0/1921537518 >> 172.21.15.23:0/1921537518 conn(0x7f40800fb870 msgr2=0x7f40800fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 -- 172.21.15.23:0/1921537518 shutdown_connections 2023-11-17T00:00:39.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.315+0000 7f40885ff700 1 -- 172.21.15.23:0/1921537518 wait complete. 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 Processor -- start 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 -- start start 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 0x7f40800748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f4080073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080073bc0 con 0x7f4080101480 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080073d30 con 0x7f4080101e10 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.316+0000 7f40885ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40801a00f0 con 0x7f4080100290 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4086b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f4080073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f408639b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 0x7f40800748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54660/0 (socket says 172.21.15.23:54660) 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 -- 172.21.15.23:0/1548387287 learned_addr learned my addr 172.21.15.23:0/1548387287 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 -- 172.21.15.23:0/1548387287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 msgr2=0x7f40800748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 0x7f40800748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 -- 172.21.15.23:0/1548387287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 msgr2=0x7f4080073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f4080073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 -- 172.21.15.23:0/1548387287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4078009d00 con 0x7f4080101480 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4086b9c700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f4080073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f4085b9a700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9a0 tx=0x7f407000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.317+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070009790 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.318+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4070010430 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.318+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408010bc70 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.318+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070018680 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.318+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408010bae0 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.319+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4080107770 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.321+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f40700105a0 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.322+0000 7f40777fe700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 0x7f406c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.322+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4070014070 con 0x7f4080101480 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.322+0000 7f408639b700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 0x7f406c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.323+0000 7f408639b700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 0x7f406c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40801012e0 tx=0x7f407c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.327+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4070051730 con 0x7f4080101480 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3797995568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1572900817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:39 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1333324320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3797995568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1572900817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:39 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1333324320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3797995568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:00:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1572900817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:39 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1333324320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:00:39.584 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.582+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4080061700 con 0x7f4080101480 2023-11-17T00:00:39.584 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.583+0000 7f40777fe700 1 -- 172.21.15.23:0/1548387287 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7f40700d0030 con 0x7f4080101480 2023-11-17T00:00:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:39.585 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 msgr2=0x7f406c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 0x7f406c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40801012e0 tx=0x7f407c009260 comp rx=0 tx=0).stop 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 msgr2=0x7f4080072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9a0 tx=0x7f407000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 shutdown_connections 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4080101e10 0x7f4080073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f406c0a11e0 0x7f406c0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080100290 0x7f40800748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 --2- 172.21.15.23:0/1548387287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4080101480 0x7f4080072ee0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 >> 172.21.15.23:0/1548387287 conn(0x7f40800fb870 msgr2=0x7f40801048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 shutdown_connections 2023-11-17T00:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.586+0000 7f40885ff700 1 -- 172.21.15.23:0/1548387287 wait complete. 2023-11-17T00:00:39.591 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:00:39.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T23:58:46.253225+0000', 'modified': '2023-11-16T23:59:34.708319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24572, 'mds_1': 24565, 'mds_2': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/3543343885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 3543343885}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 3543343885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24529': {'gid': 24529, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.173:6835/405519546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 405519546}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 405519546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/180108175', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 180108175}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 180108175}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24541': {'gid': 24541, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/1936617634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1936617634}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1936617634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24565': {'gid': 24565, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.173:6839/3306732798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 3306732798}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 3306732798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6839/1637342449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1637342449}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1637342449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-17T00:00:39.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-17T00:00:39.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-11-17T00:00:39.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/2639131328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8101420 msgr2=0x7fc4d8101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/2639131328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8101420 0x7fc4d8101870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009990 tx=0x7fc4c0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/2639131328 shutdown_connections 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/2639131328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 0x7fc4d8108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/2639131328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8100230 0x7fc4d8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/2639131328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8101420 0x7fc4d8101870 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/2639131328 >> 172.21.15.23:0/2639131328 conn(0x7fc4d80fb810 msgr2=0x7fc4d80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/2639131328 shutdown_connections 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.814+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/2639131328 wait complete. 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 Processor -- start 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 -- start start 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 0x7fc4d8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 0x7fc4d819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d819fd60 con 0x7fc4d8100230 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.815+0000 7fc4dd3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d819fea0 con 0x7fc4d8101db0 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4dd3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81a0030 con 0x7fc4d8101420 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54674/0 (socket says 172.21.15.23:54674) 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/1926798080 learned_addr learned my addr 172.21.15.23:0/1926798080 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4cffff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 0x7fc4d8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 0x7fc4d819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/1926798080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 msgr2=0x7fc4d8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 0x7fc4d8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/1926798080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 msgr2=0x7fc4d819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 0x7fc4d819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/1926798080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00095f0 con 0x7fc4d8100230 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4cffff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 0x7fc4d8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.816+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e970 tx=0x7fc4c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.817+0000 7fc4d4ff9700 1 -- 172.21.15.23:0/1926798080 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8009790 con 0x7fc4d8100230 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.817+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d810bc00 con 0x7fc4d8100230 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.817+0000 7fc4d4ff9700 1 -- 172.21.15.23:0/1926798080 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c800eec0 con 0x7fc4d8100230 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.817+0000 7fc4d4ff9700 1 -- 172.21.15.23:0/1926798080 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8003cf0 con 0x7fc4d8100230 2023-11-17T00:00:39.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.817+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d810ba70 con 0x7fc4d8100230 2023-11-17T00:00:39.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.820+0000 7fc4d4ff9700 1 -- 172.21.15.23:0/1926798080 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc4c800ccd0 con 0x7fc4d8100230 2023-11-17T00:00:39.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.822+0000 7fc4f0db3b80 1 Processor -- start 2023-11-17T00:00:39.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.822+0000 7fc4d4ff9700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 0x7fc4c40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.822+0000 7fc4f0db3b80 1 -- start start 2023-11-17T00:00:39.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4d4ff9700 1 -- 172.21.15.23:0/1926798080 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc4c8014070 con 0x7fc4d8100230 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 0x7fc4c40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f834dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f834eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f834f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f8253010 con 0x5641f834f0c0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f82a5850 con 0x5641f834e500 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f829ece0 con 0x5641f834dbb0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f834eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4d77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f834f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f834dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f834eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:44648/0 (socket says 172.21.15.23:44648) 2023-11-17T00:00:39.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.823+0000 7fc4cffff700 1 -- 172.21.15.23:0/2327005446 learned_addr learned my addr 172.21.15.23:0/2327005446 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 0x7fc4c40a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00076f0 tx=0x7fc4d0007ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/2327005446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 msgr2=0x5641f834eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f834eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/2327005446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 msgr2=0x5641f834f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f834f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/2327005446 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641f81205e0 con 0x5641f834dbb0 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f834f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f834dfc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80ca600 tx=0x7fc4c8099e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=defe97d73d4ea005 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4cd7fa700 1 -- 172.21.15.23:0/2327005446 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8099f00 con 0x5641f834dbb0 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4cd7fa700 1 -- 172.21.15.23:0/2327005446 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c809df30 con 0x5641f834dbb0 2023-11-17T00:00:39.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.824+0000 7fc4cd7fa700 1 -- 172.21.15.23:0/2327005446 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80993f0 con 0x5641f834dbb0 2023-11-17T00:00:39.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2327005446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 msgr2=0x5641f834dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f834dfc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80ca600 tx=0x7fc4c8099e00 comp rx=0 tx=0).stop 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2327005446 shutdown_connections 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f834f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f834dfc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2327005446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f834eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2327005446 >> 172.21.15.23:0/2327005446 conn(0x5641f8343310 msgr2=0x5641f8348620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2327005446 shutdown_connections 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.825+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2327005446 wait complete. 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 -- start start 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 Processor -- start 2023-11-17T00:00:39.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f8357af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834f0c0 0x5641f835eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f8240890 con 0x5641f834dbb0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f8240530 con 0x5641f834e500 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f81228d0 con 0x5641f834f0c0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4f0db3b80 10 client.0 _ll_register_callbacks cb 0x5641f825e730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54690/0 (socket says 172.21.15.23:54690) 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.826+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3093121985 learned_addr learned my addr 172.21.15.23:0/3093121985 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4cffff700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f8357af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834f0c0 0x5641f835eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834f0c0 msgr2=0x5641f835eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834f0c0 0x5641f835eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3093121985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 msgr2=0x5641f8357af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f8357af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3093121985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d810ba70 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005950 tx=0x7fc4c800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6238e2cb80b9c90 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4be7fc700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8099f00 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc4c80b81f0 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.827+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641f8340ad0 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4c80a1ec0 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 10 client.15111 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5641f8265140 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5641f8340ad0 con 0x5641f834dbb0 2023-11-17T00:00:39.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc4c8098f80 con 0x5641f834dbb0 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 10 client.15111 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 10 client.15111 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 20 client.15111 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f8362460 0x5641f8361a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 4 client.15111 mds_command: new command op to 24529 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.828+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 --> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641f8261050 con 0x5641f8362460 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.829+0000 7fc4cffff700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f8362460 0x5641f8361a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.829+0000 7fc4cffff700 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f8362460 0x5641f8361a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b26e753c1a03e61 server_cookie=1ca7b2cf98fa7f2c in_seq=0 out_seq=0 2023-11-17T00:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.829+0000 7fc4be7fc700 10 client.15111 ms_handle_connect on v2:172.21.15.173:6834/405519546 2023-11-17T00:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4be7fc700 1 -- 172.21.15.23:0/3093121985 <== mds.2 v2:172.21.15.173:6834/405519546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5641f8261050 con 0x5641f8362460 2023-11-17T00:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4be7fc700 10 client.15111 handle_command_reply: tid=0 2023-11-17T00:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 client.15111 shutdown 2023-11-17T00:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 msgr2=0x5641f83575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005950 tx=0x7fc4c800cbd0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 shutdown_connections 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834f0c0 0x5641f835eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f8362460 0x5641f8361a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834dbb0 0x5641f83575b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3093121985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f8357af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4be7fc700 0 client.15111 ms_handle_reset on v2:172.21.15.173:6834/405519546 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 >> 172.21.15.23:0/3093121985 conn(0x5641f8343310 msgr2=0x5641f8347d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 shutdown_connections 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.830+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3093121985 wait complete. 2023-11-17T00:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.831+0000 7fc4f0db3b80 20 client.15111 trim_cache size 0 max 16384 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 Processor -- start 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 -- start start 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f835eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f8351fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f8362170 con 0x5641f834f0c0 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f8248730 con 0x5641f834e500 2023-11-17T00:00:39.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.849+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f81aaed0 con 0x5641f834dbb0 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f8351fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52274/0 (socket says 172.21.15.23:52274) 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3764124777 learned_addr learned my addr 172.21.15.23:0/3764124777 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4cffff700 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f835eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3764124777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 msgr2=0x5641f835eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f835eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3764124777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 msgr2=0x5641f8351fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f8351fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/3764124777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641f8264e90 con 0x5641f834dbb0 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800c5b0 tx=0x7fc4c800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=126dc5317fa3a95d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.850+0000 7fc4cdffb700 1 -- 172.21.15.23:0/3764124777 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8005710 con 0x5641f834dbb0 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4cdffb700 1 -- 172.21.15.23:0/3764124777 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c8099f00 con 0x5641f834dbb0 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4cdffb700 1 -- 172.21.15.23:0/3764124777 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c809b180 con 0x5641f834dbb0 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3764124777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 msgr2=0x5641f83562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800c5b0 tx=0x7fc4c800cc40 comp rx=0 tx=0).stop 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3764124777 shutdown_connections 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f8351fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834dbb0 0x5641f83562a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/3764124777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834e500 0x5641f835eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3764124777 >> 172.21.15.23:0/3764124777 conn(0x5641f8343310 msgr2=0x5641f8361a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3764124777 shutdown_connections 2023-11-17T00:00:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.851+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/3764124777 wait complete. 2023-11-17T00:00:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 -- start start 2023-11-17T00:00:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 Processor -- start 2023-11-17T00:00:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 0x5641f8359d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f835a240 con 0x5641f834f0c0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f835a3b0 con 0x5641f834dbb0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f83b6800 con 0x5641f834e500 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.852+0000 7fc4f0db3b80 10 client.0 _ll_register_callbacks cb 0x7fc4d0009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 0x5641f8359d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:44674/0 (socket says 172.21.15.23:44674) 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 0x5641f8359d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52288/0 (socket says 172.21.15.23:52288) 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d6ffd700 1 -- 172.21.15.23:0/2402359496 learned_addr learned my addr 172.21.15.23:0/2402359496 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54718/0 (socket says 172.21.15.23:54718) 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 -- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 msgr2=0x5641f8359d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 0x5641f8359d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 -- 172.21.15.23:0/2402359496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 msgr2=0x5641f83597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 -- 172.21.15.23:0/2402359496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641f8265140 con 0x5641f834f0c0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4d77fe700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009fd0 tx=0x7fc4d0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf90af4909f31a server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.853+0000 7fc4be7fc700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000e050 con 0x5641f834f0c0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc4d0025450 con 0x5641f834f0c0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c809df30 con 0x5641f834f0c0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d0025610 con 0x5641f834f0c0 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 10 client.15117 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5641f80b5c10 con 0x5641f834f0c0 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c809df30 con 0x5641f834f0c0 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc4d002b020 con 0x5641f834f0c0 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 10 client.15117 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 10 client.15117 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 20 client.15117 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f83bb600 0x5641f83b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 4 client.15117 mds_command: new command op to 24529 tid=0[{"prefix": "status"}] 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.854+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 --> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5641f826d610 con 0x5641f83bb600 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.855+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f83bb600 0x5641f83b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:39.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.855+0000 7fc4d6ffd700 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f83bb600 0x5641f83b7f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bae3a2d28221953d server_cookie=192891a68842a916 in_seq=0 out_seq=0 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.855+0000 7fc4be7fc700 10 client.15117 ms_handle_connect on v2:172.21.15.173:6834/405519546 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4be7fc700 1 -- 172.21.15.23:0/2402359496 <== mds.2 v2:172.21.15.173:6834/405519546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5641f826d610 con 0x5641f83bb600 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4be7fc700 10 client.15117 handle_command_reply: tid=0 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 client.15117 shutdown 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 msgr2=0x5641f83b62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009fd0 tx=0x7fc4d0004b40 comp rx=0 tx=0).stop 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 shutdown_connections 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5641f83bb600 0x5641f83b7f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5641f834f0c0 0x5641f83b62c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641f834dbb0 0x5641f83597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4be7fc700 0 client.15117 ms_handle_reset on v2:172.21.15.173:6834/405519546 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 --2- 172.21.15.23:0/2402359496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f834e500 0x5641f8359d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 >> 172.21.15.23:0/2402359496 conn(0x5641f8343310 msgr2=0x5641f8351fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 shutdown_connections 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 1 -- 172.21.15.23:0/2402359496 wait complete. 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.856+0000 7fc4f0db3b80 20 client.15117 trim_cache size 0 max 16384 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2023-11-17T00:00:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24529, 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 66.152277745999996, 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 134.252443075 2023-11-17T00:00:39.864 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 msgr2=0x7fc4c40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 0x7fc4c40a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00076f0 tx=0x7fc4d0007ab0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 msgr2=0x7fc4d81981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e970 tx=0x7fc4c800ec80 comp rx=0 tx=0).stop 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 shutdown_connections 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101db0 0x7fc4d819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8100230 0x7fc4d81981e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc4c40a1260 0x7fc4c40a3710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 --2- 172.21.15.23:0/1926798080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4d8101420 0x7fc4d8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 >> 172.21.15.23:0/1926798080 conn(0x7fc4d80fb810 msgr2=0x7fc4d80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 shutdown_connections 2023-11-17T00:00:39.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:39.857+0000 7fc4dd3ff700 1 -- 172.21.15.23:0/1926798080 wait complete. 2023-11-17T00:00:39.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/57116007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 msgr2=0x7f0ea4101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/57116007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea4101870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009990 tx=0x7f0e98009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/57116007 shutdown_connections 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/57116007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4101db0 0x7f0ea4108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/57116007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4100230 0x7f0ea4100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/57116007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea4101870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/57116007 >> 172.21.15.23:0/57116007 conn(0x7f0ea40fb810 msgr2=0x7f0ea40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/57116007 shutdown_connections 2023-11-17T00:00:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.100+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/57116007 wait complete. 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 Processor -- start 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 -- start start 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 0x7f0ea4198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101db0 0x7f0ea419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea419fcf0 con 0x7f0ea4101db0 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea419fe30 con 0x7f0ea4100230 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0eaa9ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea419ffc0 con 0x7f0ea4101420 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 0x7f0ea4198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea8f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101db0 0x7f0ea419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52312/0 (socket says 172.21.15.23:52312) 2023-11-17T00:00:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 0x7f0ea4198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:44696/0 (socket says 172.21.15.23:44696) 2023-11-17T00:00:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.102+0000 7f0ea37fe700 1 -- 172.21.15.23:0/3446524011 learned_addr learned my addr 172.21.15.23:0/3446524011 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 -- 172.21.15.23:0/3446524011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 msgr2=0x7f0ea4198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 0x7f0ea4198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 -- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101db0 msgr2=0x7f0ea419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101db0 0x7f0ea419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 -- 172.21.15.23:0/3446524011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980095f0 con 0x7f0ea4101420 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e98004730 tx=0x7f0e98004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea17fa700 1 -- 172.21.15.23:0/3446524011 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9801d070 con 0x7f0ea4101420 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea17fa700 1 -- 172.21.15.23:0/3446524011 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e98004c90 con 0x7f0ea4101420 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea410bb80 con 0x7f0ea4101420 2023-11-17T00:00:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0ea17fa700 1 -- 172.21.15.23:0/3446524011 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980054b0 con 0x7f0ea4101420 2023-11-17T00:00:40.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.103+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea410b9f0 con 0x7f0ea4101420 2023-11-17T00:00:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.107+0000 7f0ea17fa700 1 -- 172.21.15.23:0/3446524011 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0e98005630 con 0x7f0ea4101420 2023-11-17T00:00:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ebe3d4b80 1 Processor -- start 2023-11-17T00:00:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ea17fa700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 0x7f0e8c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ebe3d4b80 1 -- start start 2023-11-17T00:00:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ea17fa700 1 -- 172.21.15.23:0/3446524011 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e980cdfa0 con 0x7f0ea4101420 2023-11-17T00:00:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 0x7f0e8c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.108+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b08511cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08511d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b085021010 con 0x55b08511bbb0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b085073850 con 0x55b08511d0c0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08506cce0 con 0x55b08511c500 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54740/0 (socket says 172.21.15.23:54740) 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 -- 172.21.15.23:0/2311369697 learned_addr learned my addr 172.21.15.23:0/2311369697 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b08511cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08511d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 -- 172.21.15.23:0/2311369697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 msgr2=0x55b08511cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b08511cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 0x7f0e8c0a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c003f20 tx=0x7f0e9c009ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 -- 172.21.15.23:0/2311369697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 msgr2=0x55b08511d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08511d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.109+0000 7f0ea3fff700 1 -- 172.21.15.23:0/2311369697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b084eee5e0 con 0x55b08511bbb0 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009d30 tx=0x7f0e94011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62fca675e66e49c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0e8a7fc700 1 -- 172.21.15.23:0/2311369697 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94004220 con 0x55b08511bbb0 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0e8a7fc700 1 -- 172.21.15.23:0/2311369697 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e94004380 con 0x55b08511bbb0 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0e8a7fc700 1 -- 172.21.15.23:0/2311369697 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e940198d0 con 0x55b08511bbb0 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2311369697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 msgr2=0x55b08511bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.110+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009d30 tx=0x7f0e94011720 comp rx=0 tx=0).stop 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2311369697 shutdown_connections 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08511d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b08511bfc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2311369697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b08511cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2311369697 >> 172.21.15.23:0/2311369697 conn(0x55b085111310 msgr2=0x55b085116620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2311369697 shutdown_connections 2023-11-17T00:00:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2311369697 wait complete. 2023-11-17T00:00:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.111+0000 7f0ebe3d4b80 1 -- start start 2023-11-17T00:00:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 Processor -- start 2023-11-17T00:00:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511bbb0 0x55b0851255c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511d0c0 0x55b08512cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08500e890 con 0x55b08511c500 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08500e530 con 0x55b08511bbb0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084ef08d0 con 0x55b08511d0c0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.112+0000 7f0ebe3d4b80 10 client.0 _ll_register_callbacks cb 0x55b08502c730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54750/0 (socket says 172.21.15.23:54750) 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 -- 172.21.15.23:0/2681032456 learned_addr learned my addr 172.21.15.23:0/2681032456 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511bbb0 0x55b0851255c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511d0c0 0x55b08512cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 -- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511d0c0 msgr2=0x55b08512cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511d0c0 0x55b08512cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 -- 172.21.15.23:0/2681032456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511bbb0 msgr2=0x55b0851255c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511bbb0 0x55b0851255c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 -- 172.21.15.23:0/2681032456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e94009960 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e98097470 tx=0x7f0e9800fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52071f49db134421 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.113+0000 7f0e837fe700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980b2070 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b08510ead0 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e980a3330 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e980b9170 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 10 client.15129 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b085033140 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e980c9080 con 0x55b08511c500 2023-11-17T00:00:40.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e9809a460 con 0x55b08511c500 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 10 client.15129 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 10 client.15129 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 20 client.15129 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085130420 0x55b08512fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 4 client.15129 mds_command: new command op to 24565 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.114+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 --> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b08502f050 con 0x55b085130420 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.115+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085130420 0x55b08512fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.115+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085130420 0x55b08512fa70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d327765abafc3063 server_cookie=634cfe251a50558 in_seq=0 out_seq=0 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.115+0000 7f0e837fe700 10 client.15129 ms_handle_connect on v2:172.21.15.173:6838/3306732798 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0e837fe700 1 -- 172.21.15.23:0/2681032456 <== mds.1 v2:172.21.15.173:6838/3306732798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b08502f050 con 0x55b085130420 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0e837fe700 10 client.15129 handle_command_reply: tid=0 2023-11-17T00:00:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 client.15129 shutdown 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 msgr2=0x55b085125b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e98097470 tx=0x7f0e9800fea0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 shutdown_connections 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085130420 0x55b08512fa70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511d0c0 0x55b08512cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511bbb0 0x55b0851255c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/2681032456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b085125b00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0e837fe700 0 client.15129 ms_handle_reset on v2:172.21.15.173:6838/3306732798 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 >> 172.21.15.23:0/2681032456 conn(0x55b085111310 msgr2=0x55b085115d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 shutdown_connections 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.116+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/2681032456 wait complete. 2023-11-17T00:00:40.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.117+0000 7f0ebe3d4b80 20 client.15129 trim_cache size 0 max 16384 2023-11-17T00:00:40.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 Processor -- start 2023-11-17T00:00:40.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 -- start start 2023-11-17T00:00:40.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 0x55b08512fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b085123460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084f78ed0 con 0x55b08511c500 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b085008e10 con 0x55b08511d0c0 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.135+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b084ef2a30 con 0x55b08511bbb0 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54756/0 (socket says 172.21.15.23:54756) 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 -- 172.21.15.23:0/997302745 learned_addr learned my addr 172.21.15.23:0/997302745 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b085123460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 0x55b08512fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 -- 172.21.15.23:0/997302745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 msgr2=0x55b08512fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 0x55b08512fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 -- 172.21.15.23:0/997302745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 msgr2=0x55b085123460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b085123460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 -- 172.21.15.23:0/997302745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b085032e90 con 0x55b08511c500 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 0x55b08512fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b085123460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e9809af00 tx=0x7f0e9809f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f51ec013986fce server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0e8affd700 1 -- 172.21.15.23:0/997302745 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980b2070 con 0x55b08511c500 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0e8affd700 1 -- 172.21.15.23:0/997302745 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b08510ead0 con 0x55b08511c500 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.136+0000 7f0e8affd700 1 -- 172.21.15.23:0/997302745 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9809af70 con 0x55b08511c500 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/997302745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 msgr2=0x55b08512cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e9809af00 tx=0x7f0e9809f120 comp rx=0 tx=0).stop 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/997302745 shutdown_connections 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b085123460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511bbb0 0x55b08512fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/997302745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511c500 0x55b08512cb00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/997302745 >> 172.21.15.23:0/997302745 conn(0x55b085111310 msgr2=0x55b085139170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/997302745 shutdown_connections 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.137+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/997302745 wait complete. 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 -- start start 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 Processor -- start 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b0851276d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08512c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08512cb30 con 0x55b08511bbb0 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08512cca0 con 0x55b08511d0c0 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b08512ce60 con 0x55b08511c500 2023-11-17T00:00:40.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ebe3d4b80 10 client.0 _ll_register_callbacks cb 0x7f0e98004e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.138+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54762/0 (socket says 172.21.15.23:54762) 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 -- 172.21.15.23:0/3066787318 learned_addr learned my addr 172.21.15.23:0/3066787318 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b0851276d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08512c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 -- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 msgr2=0x55b0851276d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b0851276d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 -- 172.21.15.23:0/3066787318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 msgr2=0x55b08512c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08512c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 -- 172.21.15.23:0/3066787318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b085033140 con 0x55b08511bbb0 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b0851276d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea8f9c700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08512c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:40.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0ea3fff700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009c50 tx=0x7f0e940044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e2ba7b50f9cd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0e837fe700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.139+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94029040 con 0x55b08511bbb0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e94004380 con 0x55b08511bbb0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e9402a440 con 0x55b08511bbb0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e9402a600 con 0x55b08511bbb0 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 10 client.15135 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:40.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b084e83c10 con 0x55b08511bbb0 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e94004380 con 0x55b08511bbb0 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e94011440 con 0x55b08511bbb0 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 10 client.15135 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 10 client.15135 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 20 client.15135 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085188db0 0x55b085185740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 4 client.15135 mds_command: new command op to 24565 tid=0[{"prefix": "status"}] 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.140+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 --> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b08503b610 con 0x55b085188db0 2023-11-17T00:00:40.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085188db0 0x55b085185740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0ea37fe700 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085188db0 0x55b085185740 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=131e7def21420959 server_cookie=5f15720ff59290c1 in_seq=0 out_seq=0 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0e837fe700 10 client.15135 ms_handle_connect on v2:172.21.15.173:6838/3306732798 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0e837fe700 1 -- 172.21.15.23:0/3066787318 <== mds.1 v2:172.21.15.173:6838/3306732798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55b08503b610 con 0x55b085188db0 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0e837fe700 10 client.15135 handle_command_reply: tid=0 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.141+0000 7f0ebe3d4b80 1 client.15135 shutdown 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 msgr2=0x55b085127190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009c50 tx=0x7f0e940044f0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 shutdown_connections 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x55b085188db0 0x55b085185740 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b08511d0c0 0x55b08512c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b08511bbb0 0x55b085127190 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 --2- 172.21.15.23:0/3066787318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b08511c500 0x55b0851276d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 >> 172.21.15.23:0/3066787318 conn(0x55b085111310 msgr2=0x55b08512fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0e837fe700 0 client.15135 ms_handle_reset on v2:172.21.15.173:6838/3306732798 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 shutdown_connections 2023-11-17T00:00:40.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 1 -- 172.21.15.23:0/3066787318 wait complete. 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.142+0000 7f0ebe3d4b80 20 client.15135 trim_cache size 0 max 16384 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24565, 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:00:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 67.419383194999995, 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:00:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 127.81889006199999 2023-11-17T00:00:40.154 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:00:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 msgr2=0x7f0e8c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 0x7f0e8c0a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c003f20 tx=0x7f0e9c009ba0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 msgr2=0x7f0ea41986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0e98004730 tx=0x7f0e98004760 comp rx=0 tx=0).stop 2023-11-17T00:00:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 shutdown_connections 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101db0 0x7f0ea419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ea4100230 0x7f0ea4198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0e8c0a1260 0x7f0e8c0a3710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 --2- 172.21.15.23:0/3446524011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ea4101420 0x7f0ea41986b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 >> 172.21.15.23:0/3446524011 conn(0x7f0ea40fb810 msgr2=0x7f0ea4106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 shutdown_connections 2023-11-17T00:00:40.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.143+0000 7f0eaa9ff700 1 -- 172.21.15.23:0/3446524011 wait complete. 2023-11-17T00:00:40.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:40 smithi023 ceph-mon[97824]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:40.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1548387287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:40.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-17T00:00:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:40 smithi173 ceph-mon[131534]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:40.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:40 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1548387287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:40.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:40 smithi156 ceph-mon[131105]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2023-11-17T00:00:40.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:40 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1548387287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:00:40.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.365+0000 7f2e933ff700 1 -- 172.21.15.23:0/2254848316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c101420 msgr2=0x7f2e8c101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.365+0000 7f2e933ff700 1 --2- 172.21.15.23:0/2254848316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c101420 0x7f2e8c101870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009990 tx=0x7f2e7c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 -- 172.21.15.23:0/2254848316 shutdown_connections 2023-11-17T00:00:40.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 --2- 172.21.15.23:0/2254848316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 0x7f2e8c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 --2- 172.21.15.23:0/2254848316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c100230 0x7f2e8c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 --2- 172.21.15.23:0/2254848316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c101420 0x7f2e8c101870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 -- 172.21.15.23:0/2254848316 >> 172.21.15.23:0/2254848316 conn(0x7f2e8c0fb810 msgr2=0x7f2e8c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 -- 172.21.15.23:0/2254848316 shutdown_connections 2023-11-17T00:00:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.366+0000 7f2e933ff700 1 -- 172.21.15.23:0/2254848316 wait complete. 2023-11-17T00:00:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 Processor -- start 2023-11-17T00:00:40.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 -- start start 2023-11-17T00:00:40.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 0x7f2e8c195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 0x7f2e8c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 0x7f2e8c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c19da90 con 0x7f2e8c101420 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c19dbd0 con 0x7f2e8c101db0 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.367+0000 7f2e933ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c19dd60 con 0x7f2e8c100230 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 0x7f2e8c195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 0x7f2e8c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 0x7f2e8c195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52358/0 (socket says 172.21.15.23:52358) 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9119b700 1 -- 172.21.15.23:0/629807718 learned_addr learned my addr 172.21.15.23:0/629807718 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9199c700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 0x7f2e8c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 -- 172.21.15.23:0/629807718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 msgr2=0x7f2e8c195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 0x7f2e8c195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 -- 172.21.15.23:0/629807718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 msgr2=0x7f2e8c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 0x7f2e8c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 -- 172.21.15.23:0/629807718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0095f0 con 0x7f2e8c101420 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.368+0000 7f2e9099a700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 0x7f2e8c196450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0059d0 tx=0x7f2e7c004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.369+0000 7f2e827fc700 1 -- 172.21.15.23:0/629807718 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c01d070 con 0x7f2e8c101420 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.369+0000 7f2e827fc700 1 -- 172.21.15.23:0/629807718 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e7c022470 con 0x7f2e8c101420 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.369+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c10bb40 con 0x7f2e8c101420 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.369+0000 7f2e827fc700 1 -- 172.21.15.23:0/629807718 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c017650 con 0x7f2e8c101420 2023-11-17T00:00:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.369+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c10b9b0 con 0x7f2e8c101420 2023-11-17T00:00:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.372+0000 7f2e827fc700 1 -- 172.21.15.23:0/629807718 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2e7c0225e0 con 0x7f2e8c101420 2023-11-17T00:00:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.373+0000 7f2ea6d41b80 1 Processor -- start 2023-11-17T00:00:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.373+0000 7f2e827fc700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.373+0000 7f2ea6d41b80 1 -- start start 2023-11-17T00:00:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.373+0000 7f2e827fc700 1 -- 172.21.15.23:0/629807718 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e7c0cdff0 con 0x7f2e8c101420 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9199c700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed23ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed24530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec28010 con 0x56041ed22bb0 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec7a850 con 0x56041ed240c0 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec73ce0 con 0x56041ed23500 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54802/0 (socket says 172.21.15.23:54802) 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 -- 172.21.15.23:0/3969454528 learned_addr learned my addr 172.21.15.23:0/3969454528 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9099a700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed23ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9199c700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed24530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 -- 172.21.15.23:0/3969454528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 msgr2=0x56041ed23ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed23ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 -- 172.21.15.23:0/3969454528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 msgr2=0x56041ed24530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed24530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9119b700 1 -- 172.21.15.23:0/3969454528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56041eaf55e0 con 0x56041ed22bb0 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9099a700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed23ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.374+0000 7f2e9199c700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed24530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:00:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.375+0000 7f2e9199c700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 0x7f2e780a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400f540 tx=0x7f2e84009920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.375+0000 7f2e9119b700 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009df0 tx=0x7f2e88011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d70f1b94e5a3929 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.375+0000 7f2e777fe700 1 -- 172.21.15.23:0/3969454528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88004230 con 0x56041ed22bb0 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.375+0000 7f2e777fe700 1 -- 172.21.15.23:0/3969454528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e88004390 con 0x56041ed22bb0 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.375+0000 7f2e777fe700 1 -- 172.21.15.23:0/3969454528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88019860 con 0x56041ed22bb0 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/3969454528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 msgr2=0x56041ed22fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009df0 tx=0x7f2e88011720 comp rx=0 tx=0).stop 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/3969454528 shutdown_connections 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed24530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed22bb0 0x56041ed22fc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/3969454528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed23ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/3969454528 >> 172.21.15.23:0/3969454528 conn(0x56041ed18310 msgr2=0x56041ed1d620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/3969454528 shutdown_connections 2023-11-17T00:00:40.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/3969454528 wait complete. 2023-11-17T00:00:40.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.376+0000 7f2ea6d41b80 1 -- start start 2023-11-17T00:00:40.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 Processor -- start 2023-11-17T00:00:40.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 0x56041ed30ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed37f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec15890 con 0x56041ed23500 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec15530 con 0x56041ed240c0 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041eaf78d0 con 0x56041ed22bb0 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2ea6d41b80 10 client.0 _ll_register_callbacks cb 0x56041ec33730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:40.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9099a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 0x56041ed30ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9099a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 0x56041ed30ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54818/0 (socket says 172.21.15.23:54818) 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52380/0 (socket says 172.21.15.23:52380) 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9119b700 1 -- 172.21.15.23:0/681792573 learned_addr learned my addr 172.21.15.23:0/681792573 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.377+0000 7f2e9199c700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed37f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 -- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 msgr2=0x56041ed37f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed37f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 -- 172.21.15.23:0/681792573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 msgr2=0x56041ed30ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 0x56041ed30ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 -- 172.21.15.23:0/681792573 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e88009a50 con 0x56041ed22bb0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e9119b700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011a70 tx=0x7f2e88003930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5821945ca8d8bbb server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e74ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:00:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88005460 con 0x56041ed22bb0 2023-11-17T00:00:40.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56041ed15ad0 con 0x56041ed22bb0 2023-11-17T00:00:40.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.378+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e880055c0 con 0x56041ed22bb0 2023-11-17T00:00:40.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e88003b20 con 0x56041ed22bb0 2023-11-17T00:00:40.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 10 client.25198 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56041ec3a140 con 0x56041ed22bb0 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56041ed15ad0 con 0x56041ed22bb0 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2e88021040 con 0x56041ed22bb0 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 10 client.25198 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 10 client.25198 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 20 client.25198 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed3bb10 0x56041ed3af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 4 client.25198 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.379+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56041ec36050 con 0x56041ed3bb10 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.380+0000 7f2e9099a700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed3bb10 0x56041ed3af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.380+0000 7f2e9099a700 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed3bb10 0x56041ed3af20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=678278c5af85ac02 server_cookie=acdf020216805917 in_seq=0 out_seq=0 2023-11-17T00:00:40.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.380+0000 7f2e74ff9700 10 client.25198 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2e74ff9700 1 -- 172.21.15.23:0/681792573 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56041ec36050 con 0x56041ed3bb10 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2e74ff9700 10 client.25198 handle_command_reply: tid=0 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 client.25198 shutdown 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 msgr2=0x56041ed30990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011a70 tx=0x7f2e88003930 comp rx=0 tx=0).stop 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 shutdown_connections 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed240c0 0x56041ed37f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed30990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed3bb10 0x56041ed3af20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/681792573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed23500 0x56041ed30ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2e74ff9700 0 client.25198 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:00:40.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 >> 172.21.15.23:0/681792573 conn(0x56041ed18310 msgr2=0x56041ed1cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 shutdown_connections 2023-11-17T00:00:40.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/681792573 wait complete. 2023-11-17T00:00:40.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.381+0000 7f2ea6d41b80 20 client.25198 trim_cache size 0 max 16384 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 Processor -- start 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 -- start start 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 0x56041ed3af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed37ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ed249b0 con 0x56041ed240c0 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ec36050 con 0x56041ed22bb0 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041eb7fed0 con 0x56041ed23500 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2e9199c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2e9199c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54830/0 (socket says 172.21.15.23:54830) 2023-11-17T00:00:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2e9199c700 1 -- 172.21.15.23:0/506360527 learned_addr learned my addr 172.21.15.23:0/506360527 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2e9119b700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 0x56041ed3af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.400+0000 7f2e9099a700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed37ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 -- 172.21.15.23:0/506360527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 msgr2=0x56041ed37ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed37ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 -- 172.21.15.23:0/506360527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 msgr2=0x56041ed3af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 0x56041ed3af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 -- 172.21.15.23:0/506360527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56041ec39e90 con 0x56041ed240c0 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9119b700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 0x56041ed3af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e9199c700 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e840197d0 tx=0x7f2e8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5390d605459aa99 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e77fff700 1 -- 172.21.15.23:0/506360527 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400e040 con 0x56041ed240c0 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e77fff700 1 -- 172.21.15.23:0/506360527 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e8400f040 con 0x56041ed240c0 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.401+0000 7f2e77fff700 1 -- 172.21.15.23:0/506360527 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84004890 con 0x56041ed240c0 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/506360527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 msgr2=0x56041ed2a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e840197d0 tx=0x7f2e8400b040 comp rx=0 tx=0).stop 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/506360527 shutdown_connections 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed2a460 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed22bb0 0x56041ed3af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/506360527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed23500 0x56041ed37ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/506360527 >> 172.21.15.23:0/506360527 conn(0x56041ed18310 msgr2=0x56041ed44880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/506360527 shutdown_connections 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/506360527 wait complete. 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.402+0000 7f2ea6d41b80 1 -- start start 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 Processor -- start 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed23500 0x56041ed33170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed8f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ed33740 con 0x56041ed240c0 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ed338b0 con 0x56041ed23500 2023-11-17T00:00:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56041ed8ff70 con 0x56041ed22bb0 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2ea6d41b80 10 client.0 _ll_register_callbacks cb 0x7f2e7c005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2e9119b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52396/0 (socket says 172.21.15.23:52396) 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2e9119b700 1 -- 172.21.15.23:0/1801788770 learned_addr learned my addr 172.21.15.23:0/1801788770 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2e9099a700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed23500 0x56041ed33170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.403+0000 7f2e9199c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed8f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 -- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed23500 msgr2=0x56041ed33170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed23500 0x56041ed33170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 -- 172.21.15.23:0/1801788770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 msgr2=0x56041ed8f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed8f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 -- 172.21.15.23:0/1801788770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8401e040 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e9119b700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2e88004880 tx=0x7f2e88003e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91af85246de08b08 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e74ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88004af0 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e88000e40 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.404+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e88004390 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e88005260 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 10 client.25204 fetch_fsmap learned FSMap version 24 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56041ea8ac10 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e88004390 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2e88011790 con 0x56041ed22bb0 2023-11-17T00:00:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 10 client.25204 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 10 client.25204 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 20 client.25204 populate_metadata read hostname 'smithi023' 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed949d0 0x56041ed91340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 4 client.25204 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56041ec42610 con 0x56041ed949d0 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.405+0000 7f2e9099a700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed949d0 0x56041ed91340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2e74ff9700 10 client.25204 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2e9099a700 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed949d0 0x56041ed91340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=277aa5672c183091 server_cookie=d34e10da873417e5 in_seq=0 out_seq=0 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2e74ff9700 1 -- 172.21.15.23:0/1801788770 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x56041ec42610 con 0x56041ed949d0 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2e74ff9700 10 client.25204 handle_command_reply: tid=0 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 client.25204 shutdown 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 msgr2=0x56041ed32c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2e88004880 tx=0x7f2e88003e70 comp rx=0 tx=0).stop 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 shutdown_connections 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56041ed949d0 0x56041ed91340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56041ed240c0 0x56041ed8f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56041ed22bb0 0x56041ed32c30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 --2- 172.21.15.23:0/1801788770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56041ed23500 0x56041ed33170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2e74ff9700 0 client.25204 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 >> 172.21.15.23:0/1801788770 conn(0x56041ed18310 msgr2=0x56041ed3af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.406+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 shutdown_connections 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2ea6d41b80 1 -- 172.21.15.23:0/1801788770 wait complete. 2023-11-17T00:00:40.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2ea6d41b80 20 client.25204 trim_cache size 0 max 16384 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24572, 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:00:40.432 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 114.147247353, 2023-11-17T00:00:40.433 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:00:40.433 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:00:40.433 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:00:40.433 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 119.346034598 2023-11-17T00:00:40.433 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 msgr2=0x7f2e780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 0x7f2e780a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400f540 tx=0x7f2e84009920 comp rx=0 tx=0).stop 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 msgr2=0x7f2e8c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 0x7f2e8c196450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0059d0 tx=0x7f2e7c004710 comp rx=0 tx=0).stop 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 shutdown_connections 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e8c101db0 0x7f2e8c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e8c100230 0x7f2e8c195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 --2- 172.21.15.23:0/629807718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e8c101420 0x7f2e8c196450 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:40.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 >> 172.21.15.23:0/629807718 conn(0x7f2e8c0fb810 msgr2=0x7f2e8c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:40.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 shutdown_connections 2023-11-17T00:00:40.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:40.407+0000 7f2e933ff700 1 -- 172.21.15.23:0/629807718 wait complete. 2023-11-17T00:00:40.435 INFO:tasks.cephadm:Setup complete, yielding 2023-11-17T00:00:40.435 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T00:00:40.445 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:00:40.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-17T00:00:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:42.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:42 smithi023 ceph-mon[97824]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:42.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:42 smithi173 ceph-mon[131534]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:42.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:42 smithi156 ceph-mon[131105]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 58 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 -- 172.21.15.23:0/1125380390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae8102520 msgr2=0x7faae8102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 --2- 172.21.15.23:0/1125380390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae8102520 0x7faae8102930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faad8009990 tx=0x7faad8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 -- 172.21.15.23:0/1125380390 shutdown_connections 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 --2- 172.21.15.23:0/1125380390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae81040b0 0x7faae810a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 --2- 172.21.15.23:0/1125380390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae8102520 0x7faae8102930 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 --2- 172.21.15.23:0/1125380390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8103720 0x7faae8103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.743+0000 7faaee8aa700 1 -- 172.21.15.23:0/1125380390 >> 172.21.15.23:0/1125380390 conn(0x7faae80fdb90 msgr2=0x7faae80fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:42.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.744+0000 7faaee8aa700 1 -- 172.21.15.23:0/1125380390 shutdown_connections 2023-11-17T00:00:42.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.744+0000 7faaee8aa700 1 -- 172.21.15.23:0/1125380390 wait complete. 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.745+0000 7faaee8aa700 1 Processor -- start 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.745+0000 7faaee8aa700 1 -- start start 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 0x7faae8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 0x7faae81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae81040b0 0x7faae819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae819fe00 con 0x7faae8102520 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae819ff40 con 0x7faae81040b0 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaee8aa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae81a00d0 con 0x7faae8103720 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae77fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 0x7faae81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 0x7faae8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae77fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 0x7faae81987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:50670/0 (socket says 172.21.15.23:50670) 2023-11-17T00:00:42.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae77fe700 1 -- 172.21.15.23:0/906918489 learned_addr learned my addr 172.21.15.23:0/906918489 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faaece47700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae81040b0 0x7faae819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 -- 172.21.15.23:0/906918489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 msgr2=0x7faae81987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 0x7faae81987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 -- 172.21.15.23:0/906918489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae81040b0 msgr2=0x7faae819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae81040b0 0x7faae819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.746+0000 7faae7fff700 1 -- 172.21.15.23:0/906918489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc009520 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faae7fff700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 0x7faae8198270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faad8009990 tx=0x7faad8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad801d070 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faad8021410 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad80095f0 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8017650 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.747+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae810dec0 con 0x7faae8102520 2023-11-17T00:00:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.749+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8109b20 con 0x7faae8102520 2023-11-17T00:00:42.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.750+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faad80177b0 con 0x7faae8102520 2023-11-17T00:00:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.752+0000 7faae57fa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 0x7faad00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.753+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faad80ce070 con 0x7faae8102520 2023-11-17T00:00:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.753+0000 7faae77fe700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 0x7faad00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.753+0000 7faae77fe700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 0x7faad00a3710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faadc009ef0 tx=0x7faadc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:42.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.756+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faad80973e0 con 0x7faae8102520 2023-11-17T00:00:42.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.979+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7faae8060f30 con 0x7faad00a1260 2023-11-17T00:00:42.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.980+0000 7faae57fa700 1 -- 172.21.15.23:0/906918489 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7faae8060f30 con 0x7faad00a1260 2023-11-17T00:00:42.982 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2023-11-17T00:00:42.982 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2023-11-17T00:00:42.982 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 msgr2=0x7faad00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 0x7faad00a3710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faadc009ef0 tx=0x7faadc009260 comp rx=0 tx=0).stop 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 msgr2=0x7faae8198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 0x7faae8198270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faad8009990 tx=0x7faad8009580 comp rx=0 tx=0).stop 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 shutdown_connections 2023-11-17T00:00:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faae81040b0 0x7faae819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faae8102520 0x7faae8198270 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.982+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faad00a1260 0x7faad00a3710 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.983+0000 7faaee8aa700 1 --2- 172.21.15.23:0/906918489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae8103720 0x7faae81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.983+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 >> 172.21.15.23:0/906918489 conn(0x7faae80fdb90 msgr2=0x7faae8106c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.983+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 shutdown_connections 2023-11-17T00:00:42.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:42.983+0000 7faaee8aa700 1 -- 172.21.15.23:0/906918489 wait complete. 2023-11-17T00:00:43.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-17T00:00:44.274 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:44 smithi156 ceph-mon[131105]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:44.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:44 smithi156 ceph-mon[131105]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 52 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:44 smithi173 ceph-mon[131534]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:44.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:44 smithi173 ceph-mon[131534]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 52 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:44 smithi023 ceph-mon[97824]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:44 smithi023 ceph-mon[97824]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 52 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 -- 172.21.15.23:0/3725164410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 msgr2=0x7f041c108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 --2- 172.21.15.23:0/3725164410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c108740 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f041400b0a0 tx=0x7f041400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 -- 172.21.15.23:0/3725164410 shutdown_connections 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 --2- 172.21.15.23:0/3725164410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c108740 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 --2- 172.21.15.23:0/3725164410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1002f0 0x7f041c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 --2- 172.21.15.23:0/3725164410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1014f0 0x7f041c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 -- 172.21.15.23:0/3725164410 >> 172.21.15.23:0/3725164410 conn(0x7f041c0fb900 msgr2=0x7f041c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.775+0000 7f0422332700 1 -- 172.21.15.23:0/3725164410 shutdown_connections 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.776+0000 7f0422332700 1 -- 172.21.15.23:0/3725164410 wait complete. 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.776+0000 7f0422332700 1 Processor -- start 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 -- start start 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 0x7f041c196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1014f0 0x7f041c196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c19dce0 con 0x7f041c101e80 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c19de20 con 0x7f041c1014f0 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f0422332700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c19dfb0 con 0x7f041c1002f0 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f04208cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f04208cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41752/0 (socket says 172.21.15.23:41752) 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f04208cf700 1 -- 172.21.15.23:0/1484915441 learned_addr learned my addr 172.21.15.23:0/1484915441 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f041bfff700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 0x7f041c196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.777+0000 7f041b7fe700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1014f0 0x7f041c196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 -- 172.21.15.23:0/1484915441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 msgr2=0x7f041c196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 0x7f041c196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 -- 172.21.15.23:0/1484915441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1014f0 msgr2=0x7f041c196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1014f0 0x7f041c196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 -- 172.21.15.23:0/1484915441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414009d00 con 0x7f041c101e80 2023-11-17T00:00:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f041bfff700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 0x7f041c196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04208cf700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f041400b070 tx=0x7f041400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041400e040 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c10bcb0 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0414003e10 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.778+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041401b6c0 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.779+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c10bb20 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.780+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041c107860 con 0x7f041c101e80 2023-11-17T00:00:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.782+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f041401d470 con 0x7f041c101e80 2023-11-17T00:00:45.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.783+0000 7f04197fa700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 0x7f04040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:45.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.783+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f04140cdec0 con 0x7f041c101e80 2023-11-17T00:00:45.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.783+0000 7f041bfff700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 0x7f04040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:45.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.784+0000 7f041bfff700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 0x7f04040a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f041c101350 tx=0x7f040c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:45.787+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f04140971d0 con 0x7f041c101e80 2023-11-17T00:00:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.007+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f041c061980 con 0x7f04040a1260 2023-11-17T00:00:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.021+0000 7f04197fa700 1 -- 172.21.15.23:0/1484915441 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7f041c061980 con 0x7f04040a1260 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (2m) 42s ago 2m 13.7M - 18.0.0-3906-g1c956514 ace6904b066b 60d1e85599ab 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi156 running (2m) 52s ago 2m 30.2M - 18.0.0-3906-g1c956514 ace6904b066b fdd87a7c7ab3 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi173 running (2m) 52s ago 2m 17.7M - 18.0.0-3906-g1c956514 ace6904b066b d648d773423a 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (2m) 42s ago 2m 13.9M - 18.0.0-3906-g1c956514 ace6904b066b 0312ec601765 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi156 running (2m) 52s ago 2m 19.1M - 18.0.0-3906-g1c956514 ace6904b066b ef9fcd63aa97 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi173 running (2m) 52s ago 2m 19.7M - 18.0.0-3906-g1c956514 ace6904b066b 205b78625cbd 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (2m) 42s ago 2m 13.7M - 18.0.0-3906-g1c956514 ace6904b066b 5bbb35e14b84 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi156 running (2m) 52s ago 2m 19.1M - 18.0.0-3906-g1c956514 ace6904b066b 24d30aef9aef 2023-11-17T00:00:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi173 running (2m) 52s ago 2m 18.5M - 18.0.0-3906-g1c956514 ace6904b066b 41bc786cfd48 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (2m) 42s ago 2m 13.9M - 18.0.0-3906-g1c956514 ace6904b066b 12182c2089fb 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi156 running (2m) 52s ago 2m 14.5M - 18.0.0-3906-g1c956514 ace6904b066b 181380572c0d 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi173 running (2m) 52s ago 2m 15.2M - 18.0.0-3906-g1c956514 ace6904b066b 0b61ee55a09b 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (10m) 42s ago 10m 521M - 18.0.0-3906-g1c956514 ace6904b066b fb5144615a3b 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi156 *:8443,8765 running (7m) 52s ago 7m 449M - 18.0.0-3906-g1c956514 ace6904b066b 331a0e3ea21d 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi173 *:8443,8765 running (8m) 52s ago 8m 448M - 18.0.0-3906-g1c956514 ace6904b066b 884f6816a04e 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (10m) 42s ago 10m 96.9M 2048M 18.0.0-3906-g1c956514 ace6904b066b e083aa47914f 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi156 running (8m) 52s ago 8m 65.4M 2048M 18.0.0-3906-g1c956514 ace6904b066b d62b23c6552d 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi173 running (8m) 52s ago 8m 65.2M 2048M 18.0.0-3906-g1c956514 ace6904b066b 55a30f0f87ef 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (7m) 42s ago 7m 71.6M 4317M 18.0.0-3906-g1c956514 ace6904b066b 758d501792ba 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi156 running (7m) 52s ago 7m 77.4M 4292M 18.0.0-3906-g1c956514 ace6904b066b 3f928a64ca33 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi173 running (6m) 52s ago 6m 74.7M 4292M 18.0.0-3906-g1c956514 ace6904b066b a21b9c95376f 2023-11-17T00:00:46.024 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (6m) 42s ago 6m 65.9M 4317M 18.0.0-3906-g1c956514 ace6904b066b 12a52f9f7e21 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi156 running (5m) 52s ago 5m 63.1M 4292M 18.0.0-3906-g1c956514 ace6904b066b f85633cf7e3d 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi173 running (5m) 52s ago 5m 67.9M 4292M 18.0.0-3906-g1c956514 ace6904b066b 4adb0df5305e 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (5m) 42s ago 5m 60.5M 4317M 18.0.0-3906-g1c956514 ace6904b066b b3314fd6b4b8 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi156 running (4m) 52s ago 4m 66.2M 4292M 18.0.0-3906-g1c956514 ace6904b066b 0b13b238d59e 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi173 running (4m) 52s ago 4m 62.7M 4292M 18.0.0-3906-g1c956514 ace6904b066b 4eaae953d9dd 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (3m) 42s ago 3m 58.9M 4317M 18.0.0-3906-g1c956514 ace6904b066b 53d740e976af 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi156 running (3m) 52s ago 3m 60.2M 4292M 18.0.0-3906-g1c956514 ace6904b066b 2c67f07daf4a 2023-11-17T00:00:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi173 running (2m) 52s ago 2m 56.1M 4292M 18.0.0-3906-g1c956514 ace6904b066b 2bf0900088d3 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 msgr2=0x7f04040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 0x7f04040a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f041c101350 tx=0x7f040c00b410 comp rx=0 tx=0).stop 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 msgr2=0x7f041c19d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f041400b070 tx=0x7f041400bce0 comp rx=0 tx=0).stop 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 shutdown_connections 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c101e80 0x7f041c19d710 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f04040a1260 0x7f04040a3710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f041c1002f0 0x7f041c196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 --2- 172.21.15.23:0/1484915441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f041c1014f0 0x7f041c196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.023+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 >> 172.21.15.23:0/1484915441 conn(0x7f041c0fb900 msgr2=0x7f041c1049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.024+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 shutdown_connections 2023-11-17T00:00:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:46.024+0000 7f0422332700 1 -- 172.21.15.23:0/1484915441 wait complete. 2023-11-17T00:00:46.448 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:46 smithi023 ceph-mon[97824]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:46.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-17T00:00:46.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:46 smithi173 ceph-mon[131534]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:46.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:46 smithi156 ceph-mon[131105]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:47.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:47 smithi023 ceph-mon[97824]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:47.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:47 smithi173 ceph-mon[131534]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:47.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:47 smithi156 ceph-mon[131105]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:48.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:48 smithi023 ceph-mon[97824]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:48.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:48 smithi173 ceph-mon[131534]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:48 smithi156 ceph-mon[131105]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2023-11-17T00:00:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.749+0000 7ff44c3af700 1 -- 172.21.15.23:0/1886561911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444102590 msgr2=0x7ff4441029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.749+0000 7ff44c3af700 1 --2- 172.21.15.23:0/1886561911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444102590 0x7ff4441029a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff440009990 tx=0x7ff440009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 -- 172.21.15.23:0/1886561911 shutdown_connections 2023-11-17T00:00:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 --2- 172.21.15.23:0/1886561911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444104120 0x7ff44410a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 --2- 172.21.15.23:0/1886561911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444102590 0x7ff4441029a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 --2- 172.21.15.23:0/1886561911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444103790 0x7ff444103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 -- 172.21.15.23:0/1886561911 >> 172.21.15.23:0/1886561911 conn(0x7ff4440fdba0 msgr2=0x7ff444100010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 -- 172.21.15.23:0/1886561911 shutdown_connections 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.750+0000 7ff44c3af700 1 -- 172.21.15.23:0/1886561911 wait complete. 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 Processor -- start 2023-11-17T00:00:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 -- start start 2023-11-17T00:00:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 0x7ff444198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 0x7ff44419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419fde0 con 0x7ff444104120 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419ff20 con 0x7ff444102590 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44c3af700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4441a00b0 con 0x7ff444103790 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44994a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44994a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:50712/0 (socket says 172.21.15.23:50712) 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44a14b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 0x7ff444198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44994a700 1 -- 172.21.15.23:0/195038674 learned_addr learned my addr 172.21.15.23:0/195038674 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.751+0000 7ff44a94c700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 0x7ff44419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 -- 172.21.15.23:0/195038674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 msgr2=0x7ff444198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 -- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 msgr2=0x7ff44419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 0x7ff44419f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 -- 172.21.15.23:0/195038674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400095f0 con 0x7ff444102590 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44994a700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a94c700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 0x7ff44419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff44a14b700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 0x7ff444198250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff4400098e0 tx=0x7ff440009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.752+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44001d070 con 0x7ff444102590 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.753+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff440021410 con 0x7ff444102590 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.753+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44410df40 con 0x7ff444102590 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.753+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440005720 con 0x7ff444102590 2023-11-17T00:00:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.753+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44410ddb0 con 0x7ff444102590 2023-11-17T00:00:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.754+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff444109520 con 0x7ff444102590 2023-11-17T00:00:48.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.756+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff440021580 con 0x7ff444102590 2023-11-17T00:00:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.757+0000 7ff43b7fe700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 0x7ff4300a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.757+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4400ce170 con 0x7ff444102590 2023-11-17T00:00:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.757+0000 7ff44994a700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 0x7ff4300a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.758+0000 7ff44994a700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 0x7ff4300a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff434011550 tx=0x7ff434009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.759+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff440097480 con 0x7ff444102590 2023-11-17T00:00:48.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.975+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff444061c30 con 0x7ff4300a1260 2023-11-17T00:00:48.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.981+0000 7ff43b7fe700 1 -- 172.21.15.23:0/195038674 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7ff444061c30 con 0x7ff4300a1260 2023-11-17T00:00:48.983 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-17T00:00:48.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 55s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2023-11-17T00:00:48.983 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 55s ago 8m smithi023=x;smithi156=y;smithi173=z;count:3 2023-11-17T00:00:48.983 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 55s ago 8m smithi023:172.21.15.23=a;smithi156:172.21.15.156=b;smithi173:172.21.15.173=c;count:3 2023-11-17T00:00:48.983 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 55s ago - 2023-11-17T00:00:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 msgr2=0x7ff4300a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 0x7ff4300a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff434011550 tx=0x7ff434009260 comp rx=0 tx=0).stop 2023-11-17T00:00:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 msgr2=0x7ff444198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 0x7ff444198250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff4400098e0 tx=0x7ff440009580 comp rx=0 tx=0).stop 2023-11-17T00:00:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 shutdown_connections 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff444104120 0x7ff44419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444102590 0x7ff444198250 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff4300a1260 0x7ff4300a3710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 --2- 172.21.15.23:0/195038674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff444103790 0x7ff444198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.983+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 >> 172.21.15.23:0/195038674 conn(0x7ff4440fdba0 msgr2=0x7ff4440ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.984+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 shutdown_connections 2023-11-17T00:00:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:48.984+0000 7ff44c3af700 1 -- 172.21.15.23:0/195038674 wait complete. 2023-11-17T00:00:49.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-17T00:00:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:49 smithi023 ceph-mon[97824]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:50.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:49 smithi023 ceph-mon[97824]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:49 smithi173 ceph-mon[131534]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:49 smithi173 ceph-mon[131534]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:49 smithi156 ceph-mon[131105]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:50.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:49 smithi156 ceph-mon[131105]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:50.296 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 -- 172.21.15.23:0/2378236227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1002f0 msgr2=0x7f9b0c100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2378236227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c100700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009990 tx=0x7f9afc009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 -- 172.21.15.23:0/2378236227 shutdown_connections 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2378236227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2378236227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c100700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2378236227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 -- 172.21.15.23:0/2378236227 >> 172.21.15.23:0/2378236227 conn(0x7f9b0c0fb900 msgr2=0x7f9b0c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:51.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 -- 172.21.15.23:0/2378236227 shutdown_connections 2023-11-17T00:00:51.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.743+0000 7f9b1101c700 1 -- 172.21.15.23:0/2378236227 wait complete. 2023-11-17T00:00:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 Processor -- start 2023-11-17T00:00:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 -- start start 2023-11-17T00:00:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c073ba0 con 0x7f9b0c101e80 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c073d10 con 0x7f9b0c1002f0 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.744+0000 7f9b1101c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c1a0110 con 0x7f9b0c1014f0 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:50870/0 (socket says 172.21.15.23:50870) 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 -- 172.21.15.23:0/2628853600 learned_addr learned my addr 172.21.15.23:0/2628853600 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0b59e700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0ad9d700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 -- 172.21.15.23:0/2628853600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1002f0 msgr2=0x7f9b0c074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 -- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 msgr2=0x7f9b0c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 -- 172.21.15.23:0/2628853600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0095f0 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0b59e700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.745+0000 7f9b0a59c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9a0 tx=0x7f9b0000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.746+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00009790 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.746+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b00010430 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.746+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00018680 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.746+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c10bcb0 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.746+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c10bb20 con 0x7f9b0c1014f0 2023-11-17T00:00:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.747+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c107930 con 0x7f9b0c1014f0 2023-11-17T00:00:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.749+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9b000105a0 con 0x7f9b0c1014f0 2023-11-17T00:00:51.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.750+0000 7f9af3fff700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 0x7f9af40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:51.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.751+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9b00014070 con 0x7f9b0c1014f0 2023-11-17T00:00:51.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.751+0000 7f9b0ad9d700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 0x7f9af40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.751+0000 7f9b0ad9d700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 0x7f9af40a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00b2e0 tx=0x7f9afc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.753+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9b00096220 con 0x7f9b0c1014f0 2023-11-17T00:00:51.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.966+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b0c02cfe0 con 0x7f9af40a1260 2023-11-17T00:00:51.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.967+0000 7f9af3fff700 1 -- 172.21.15.23:0/2628853600 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f9b0c02cfe0 con 0x7f9af40a1260 2023-11-17T00:00:51.970 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2023-11-17T00:00:51.970 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2023-11-17T00:00:51.970 INFO:teuthology.orchestra.run.smithi023.stdout:smithi156 172.21.15.156 2023-11-17T00:00:51.970 INFO:teuthology.orchestra.run.smithi023.stdout:smithi173 172.21.15.173 2023-11-17T00:00:51.970 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 msgr2=0x7f9af40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 0x7f9af40a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00b2e0 tx=0x7f9afc005fb0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 msgr2=0x7f9b0c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9a0 tx=0x7f9b0000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 shutdown_connections 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b0c101e80 0x7f9b0c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b0c1002f0 0x7f9b0c074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9af40a1260 0x7f9af40a3710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 --2- 172.21.15.23:0/2628853600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b0c1014f0 0x7f9b0c072ee0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:51.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 >> 172.21.15.23:0/2628853600 conn(0x7f9b0c0fb900 msgr2=0x7f9b0c104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 shutdown_connections 2023-11-17T00:00:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:51.970+0000 7f9b1101c700 1 -- 172.21.15.23:0/2628853600 wait complete. 2023-11-17T00:00:52.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:52 smithi023 ceph-mon[97824]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:52.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-17T00:00:52.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:52 smithi173 ceph-mon[131534]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:52.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:52 smithi156 ceph-mon[131105]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:00:53.360 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:53.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:53 smithi023 ceph-mon[97824]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:53.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:53 smithi173 ceph-mon[131534]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:53.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:53 smithi156 ceph-mon[131105]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:54.207 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:54 smithi023 ceph-mon[97824]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:54.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:54 smithi173 ceph-mon[131534]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:54.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:54 smithi156 ceph-mon[131105]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:54.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 -- 172.21.15.23:0/619985752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 msgr2=0x7f9160100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 --2- 172.21.15.23:0/619985752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f9160100700 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f915c009990 tx=0x7f915c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 -- 172.21.15.23:0/619985752 shutdown_connections 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 --2- 172.21.15.23:0/619985752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f9160108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 --2- 172.21.15.23:0/619985752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f9160100700 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 --2- 172.21.15.23:0/619985752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f9160101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.883+0000 7f9167391700 1 -- 172.21.15.23:0/619985752 >> 172.21.15.23:0/619985752 conn(0x7f91600fb900 msgr2=0x7f91600fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.884+0000 7f9167391700 1 -- 172.21.15.23:0/619985752 shutdown_connections 2023-11-17T00:00:54.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.884+0000 7f9167391700 1 -- 172.21.15.23:0/619985752 wait complete. 2023-11-17T00:00:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 Processor -- start 2023-11-17T00:00:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 -- start start 2023-11-17T00:00:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f91601987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f916019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916019fe30 con 0x7f91601002f0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916019ff70 con 0x7f9160101e80 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f9167391700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91601a0100 con 0x7f91601014f0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f916512d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f916512d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44406/0 (socket says 172.21.15.23:44406) 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.885+0000 7f916512d700 1 -- 172.21.15.23:0/1866603415 learned_addr learned my addr 172.21.15.23:0/1866603415 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916492c700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f91601987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916592e700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f916019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 -- 172.21.15.23:0/1866603415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 msgr2=0x7f91601987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f91601987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 -- 172.21.15.23:0/1866603415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 msgr2=0x7f916019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f916019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 -- 172.21.15.23:0/1866603415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c0095f0 con 0x7f91601002f0 2023-11-17T00:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916492c700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f91601987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916592e700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f916019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f916512d700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f915c004740 tx=0x7f915c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c01d070 con 0x7f91601002f0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.886+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916010bcb0 con 0x7f91601002f0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.887+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f915c00fbc0 con 0x7f91601002f0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.887+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c017650 con 0x7f91601002f0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.887+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916010bb20 con 0x7f91601002f0 2023-11-17T00:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.888+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9160192a10 con 0x7f91601002f0 2023-11-17T00:00:54.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.890+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f915c021410 con 0x7f91601002f0 2023-11-17T00:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.891+0000 7f91567fc700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 0x7f914c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.891+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f915c0ce370 con 0x7f91601002f0 2023-11-17T00:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.891+0000 7f916492c700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 0x7f914c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.892+0000 7f916492c700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 0x7f914c0a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9150009cb0 tx=0x7f9150009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:54.895+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f915c0976e0 con 0x7f91601002f0 2023-11-17T00:00:55.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.109+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 --> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9160061970 con 0x7f914c0a11e0 2023-11-17T00:00:55.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.111+0000 7f91567fc700 1 -- 172.21.15.23:0/1866603415 <== mgr.14150 v2:172.21.15.23:6800/499313593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f9160061970 con 0x7f914c0a11e0 2023-11-17T00:00:55.112 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-17T00:00:55.112 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 3m ago LVM detected, locked 2023-11-17T00:00:55.113 INFO:teuthology.orchestra.run.smithi023.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 3m ago LVM detected, locked 2023-11-17T00:00:55.113 INFO:teuthology.orchestra.run.smithi023.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 2m ago LVM detected, locked 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 msgr2=0x7f914c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 0x7f914c0a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9150009cb0 tx=0x7f9150009260 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 msgr2=0x7f91601982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f915c004740 tx=0x7f915c009580 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 shutdown_connections 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9160101e80 0x7f916019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91601002f0 0x7f91601982a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f914c0a11e0 0x7f914c0a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 --2- 172.21.15.23:0/1866603415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91601014f0 0x7f91601987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 >> 172.21.15.23:0/1866603415 conn(0x7f91600fb900 msgr2=0x7f9160106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 shutdown_connections 2023-11-17T00:00:55.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:55.113+0000 7f9167391700 1 -- 172.21.15.23:0/1866603415 wait complete. 2023-11-17T00:00:55.567 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T00:00:55.723 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:00:55.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-11-17T00:00:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:56 smithi023 ceph-mon[97824]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:56 smithi023 ceph-mon[97824]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:56.589 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:00:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:56 smithi156 ceph-mon[131105]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:56 smithi156 ceph-mon[131105]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:56 smithi173 ceph-mon[131534]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-17T00:00:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:56 smithi173 ceph-mon[131534]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T00:00:57.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1848990002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8074b20 msgr2=0x7fe7d8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1848990002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8074b20 0x7fe7d8072f80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0099c0 tx=0x7fe7cc009cd0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1848990002 shutdown_connections 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1848990002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d810c8d0 0x7fe7d810ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1848990002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8074b20 0x7fe7d8072f80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1848990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d8073550 0x7fe7d80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1848990002 >> 172.21.15.23:0/1848990002 conn(0x7fe7d80fb900 msgr2=0x7fe7d80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.964+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1848990002 shutdown_connections 2023-11-17T00:00:57.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.965+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1848990002 wait complete. 2023-11-17T00:00:57.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 Processor -- start 2023-11-17T00:00:57.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 -- start start 2023-11-17T00:00:57.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:57.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 0x7fe7d819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d810c8d0 0x7fe7d81a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81a4200 con 0x7fe7d8073550 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81a4340 con 0x7fe7d8074b20 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dea6b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81a44d0 con 0x7fe7d810c8d0 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dc807700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dc807700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44420/0 (socket says 172.21.15.23:44420) 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.966+0000 7fe7dc807700 1 -- 172.21.15.23:0/1653475510 learned_addr learned my addr 172.21.15.23:0/1653475510 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7d7fff700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 0x7fe7d819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dd008700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d810c8d0 0x7fe7d81a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 -- 172.21.15.23:0/1653475510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d810c8d0 msgr2=0x7fe7d81a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d810c8d0 0x7fe7d81a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 -- 172.21.15.23:0/1653475510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 msgr2=0x7fe7d819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:57.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 0x7fe7d819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:57.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 -- 172.21.15.23:0/1653475510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc0095f0 con 0x7fe7d8073550 2023-11-17T00:00:57.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7d7fff700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 0x7fe7d819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:00:57.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7dc807700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098e0 tx=0x7fe7cc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:57.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.967+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc01d070 con 0x7fe7d8073550 2023-11-17T00:00:57.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.968+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7cc005030 con 0x7fe7d8073550 2023-11-17T00:00:57.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.968+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d81122a0 con 0x7fe7d8073550 2023-11-17T00:00:57.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.968+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc004a70 con 0x7fe7d8073550 2023-11-17T00:00:57.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.968+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d8112110 con 0x7fe7d8073550 2023-11-17T00:00:57.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.969+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d8102420 con 0x7fe7d8073550 2023-11-17T00:00:57.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.972+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe7cc0051a0 con 0x7fe7d8073550 2023-11-17T00:00:57.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.974+0000 7fe7d5ffb700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 0x7fe7c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:00:57.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.974+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe7cc0cdfb0 con 0x7fe7d8073550 2023-11-17T00:00:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.974+0000 7fe7d7fff700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 0x7fe7c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:00:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.975+0000 7fe7d7fff700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 0x7fe7c80a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4011490 tx=0x7fe7c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:00:57.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:57.976+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe7cc0972c0 con 0x7fe7d8073550 2023-11-17T00:00:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.226+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe7d802cfe0 con 0x7fe7d8073550 2023-11-17T00:00:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.227+0000 7fe7d5ffb700 1 -- 172.21.15.23:0/1653475510 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2905 (secure 0 0 0) 0x7fe7cc026b70 con 0x7fe7d8073550 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:e24 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 24 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-11-17T00:00:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:created 2023-11-16T23:58:46.253225+0000 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2023-11-16T23:59:34.708319+0000 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T00:00:58.230 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 3 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1,2 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24572,1=24565,2=24529} 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3,4] 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2023-11-17T00:00:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{0:24572} state up:active seq 3 addr [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{0:24541} state up:standby-replay seq 1 addr [v2:172.21.15.173:6836/1936617634,v1:172.21.15.173:6837/1936617634] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{1:24565} state up:active seq 17 addr [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{1:24533} state up:standby-replay seq 1 addr [v2:172.21.15.156:6836/180108175,v1:172.21.15.156:6837/180108175] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{2:24529} state up:active seq 19 addr [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{2:24500} state up:standby-replay seq 1 addr [v2:172.21.15.156:6834/3543343885,v1:172.21.15.156:6835/3543343885] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14463} state up:standby seq 1 addr [v2:172.21.15.23:6834/1622872783,v1:172.21.15.23:6835/1622872783] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14475} state up:standby seq 1 addr [v2:172.21.15.23:6836/4279927842,v1:172.21.15.23:6837/4279927842] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.232 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{-1:14529} state up:standby seq 1 addr [v2:172.21.15.23:6838/3704661789,v1:172.21.15.23:6839/3704661789] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.233 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{-1:14541} state up:standby seq 1 addr [v2:172.21.15.23:6840/3035227424,v1:172.21.15.23:6841/3035227424] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.233 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{-1:24464} state up:standby seq 1 addr [v2:172.21.15.156:6832/2227062426,v1:172.21.15.156:6833/2227062426] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.233 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{-1:24472} state up:standby seq 1 addr [v2:172.21.15.173:6832/1743311827,v1:172.21.15.173:6833/1743311827] compat {c=[1],r=[1],i=[7ff]}] 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 msgr2=0x7fe7c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 0x7fe7c80a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4011490 tx=0x7fe7c4009260 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 msgr2=0x7fe7d819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098e0 tx=0x7fe7cc00f690 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 shutdown_connections 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe7c80a11e0 0x7fe7c80a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d810c8d0 0x7fe7d81a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7d8073550 0x7fe7d819c670 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 --2- 172.21.15.23:0/1653475510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7d8074b20 0x7fe7d819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 >> 172.21.15.23:0/1653475510 conn(0x7fe7d80fb900 msgr2=0x7fe7d80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.229+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 shutdown_connections 2023-11-17T00:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:00:58.230+0000 7fe7dea6b700 1 -- 172.21.15.23:0/1653475510 wait complete. 2023-11-17T00:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:00:58.491 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:58 smithi023 ceph-mon[97824]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:58 smithi173 ceph-mon[131534]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:58 smithi156 ceph-mon[131105]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:00:58.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-11-17T00:00:59.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:00:59 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1653475510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-17T00:00:59.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:00:59 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1653475510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-17T00:00:59.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:00:59 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1653475510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-17T00:00:59.650 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:01:00.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:00 smithi023 ceph-mon[97824]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:01:00.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:00 smithi173 ceph-mon[131534]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:01:00.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:00 smithi156 ceph-mon[131105]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 -- 172.21.15.23:0/598059086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 msgr2=0x7ff1a8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 --2- 172.21.15.23:0/598059086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8072f80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009990 tx=0x7ff1a4009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 -- 172.21.15.23:0/598059086 shutdown_connections 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 --2- 172.21.15.23:0/598059086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a81081c0 0x7ff1a810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 --2- 172.21.15.23:0/598059086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8072f80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 --2- 172.21.15.23:0/598059086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a8073550 0x7ff1a80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 -- 172.21.15.23:0/598059086 >> 172.21.15.23:0/598059086 conn(0x7ff1a80fb900 msgr2=0x7ff1a80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 -- 172.21.15.23:0/598059086 shutdown_connections 2023-11-17T00:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.331+0000 7ff1af75f700 1 -- 172.21.15.23:0/598059086 wait complete. 2023-11-17T00:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.332+0000 7ff1af75f700 1 Processor -- start 2023-11-17T00:01:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 -- start start 2023-11-17T00:01:01.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 0x7ff1a8198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:01.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:01.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a81081c0 0x7ff1a819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:01.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a819fdc0 con 0x7ff1a8074b20 2023-11-17T00:01:01.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a819ff00 con 0x7ff1a8073550 2023-11-17T00:01:01.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1af75f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a81a0090 con 0x7ff1a81081c0 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1accfa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1ad4fb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 0x7ff1a8198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1accfa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50374/0 (socket says 172.21.15.23:50374) 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1accfa700 1 -- 172.21.15.23:0/1172156498 learned_addr learned my addr 172.21.15.23:0/1172156498 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.333+0000 7ff1adcfc700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a81081c0 0x7ff1a819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 -- 172.21.15.23:0/1172156498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a81081c0 msgr2=0x7ff1a819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a81081c0 0x7ff1a819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 -- 172.21.15.23:0/1172156498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 msgr2=0x7ff1a8198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 0x7ff1a8198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 -- 172.21.15.23:0/1172156498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40095f0 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1ad4fb700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 0x7ff1a8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1accfa700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff19800e9d0 tx=0x7ff19800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198009790 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff19800cbd0 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198018680 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a810de90 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.334+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a810dd00 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.335+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a8106870 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.337+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff19800cd40 con 0x7ff1a8074b20 2023-11-17T00:01:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.338+0000 7ff19e7fc700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 0x7ff1940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:01.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.339+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff198014070 con 0x7ff1a8074b20 2023-11-17T00:01:01.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.339+0000 7ff1ad4fb700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 0x7ff1940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:01.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.339+0000 7ff1ad4fb700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 0x7ff1940a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400b2e0 tx=0x7ff1a4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:01.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.343+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff198051730 con 0x7ff1a8074b20 2023-11-17T00:01:01.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.550+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7ff1a8061700 con 0x7ff1a8074b20 2023-11-17T00:01:01.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.551+0000 7ff19e7fc700 1 -- 172.21.15.23:0/1172156498 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) v1 ==== 56+0+5608 (secure 0 0 0) 0x7ff198096340 con 0x7ff1a8074b20 2023-11-17T00:01:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 90 2023-11-17T00:01:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 2023-11-17T00:01:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:created 2023-11-16T23:50:32.514395+0000 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2023-11-17T00:00:32.047239+0000 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 27 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release reef 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-11-17T00:01:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/2230726464,v1:172.21.15.23:6803/2230726464] [v2:172.21.15.23:6804/2230726464,v1:172.21.15.23:6805/2230726464] exists,up a823ad49-73e7-442f-89dc-91875e102c7b 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/2641693166,v1:172.21.15.156:6801/2641693166] [v2:172.21.15.156:6802/2641693166,v1:172.21.15.156:6803/2641693166] exists,up 26bd38f3-3da3-45e1-9d3b-d375d7fa95ef 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/2331881016,v1:172.21.15.173:6801/2331881016] [v2:172.21.15.173:6802/2331881016,v1:172.21.15.173:6803/2331881016] exists,up 6b22c974-66e4-455c-882c-4804d393aec2 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/2975279435,v1:172.21.15.23:6811/2975279435] [v2:172.21.15.23:6812/2975279435,v1:172.21.15.23:6813/2975279435] exists,up 82122923-cabb-475e-ab26-d6a7bb6c88ad 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 33 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/1251331941,v1:172.21.15.156:6809/1251331941] [v2:172.21.15.156:6810/1251331941,v1:172.21.15.156:6811/1251331941] exists,up dbe06115-7a9f-4be5-b2df-3ac182c2f34f 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 38 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/2003227245,v1:172.21.15.173:6809/2003227245] [v2:172.21.15.173:6810/2003227245,v1:172.21.15.173:6811/2003227245] exists,up 5cb74da1-90da-4bfa-a528-b08833cabdcb 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/1429107943,v1:172.21.15.23:6819/1429107943] [v2:172.21.15.23:6820/1429107943,v1:172.21.15.23:6821/1429107943] exists,up 37b75c4c-cc8a-4327-bfda-9da89338654b 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 51 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/2873383539,v1:172.21.15.156:6817/2873383539] [v2:172.21.15.156:6818/2873383539,v1:172.21.15.156:6819/2873383539] exists,up 1f9a70dc-a716-42aa-a798-635dd1f18ce1 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 58 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/2703397724,v1:172.21.15.173:6817/2703397724] [v2:172.21.15.173:6818/2703397724,v1:172.21.15.173:6819/2703397724] exists,up 31074fdd-3ccc-460e-91e1-10339bf34c95 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 66 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/2722703119,v1:172.21.15.23:6827/2722703119] [v2:172.21.15.23:6828/2722703119,v1:172.21.15.23:6829/2722703119] exists,up 87031132-5932-4aa2-ad9d-4b706edfac8f 2023-11-17T00:01:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 71 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/651591322,v1:172.21.15.156:6825/651591322] [v2:172.21.15.156:6826/651591322,v1:172.21.15.156:6827/651591322] exists,up afa77aab-17f7-4ab5-a277-c10102ad7e5d 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 79 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/3375444046,v1:172.21.15.173:6825/3375444046] [v2:172.21.15.173:6826/3375444046,v1:172.21.15.173:6827/3375444046] exists,up f38c6af8-9481-4757-9973-dba7aa1a03b1 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.8 [2,11] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.c [2,4] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.1d [8,11] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.0 [4,6] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.5 [3,9] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.10 [5,9] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.11 [4,8] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.3c [4,9] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.2 [9,1] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1c [9,8] 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2730190791 expires 2023-11-17T23:51:30.957768+0000 2023-11-17T00:01:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/3289144239 expires 2023-11-17T23:51:30.957768+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/3289144239 expires 2023-11-17T23:51:30.957768+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/4236499031 expires 2023-11-17T23:51:04.390538+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/1477838149 expires 2023-11-17T23:51:04.390538+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2789478528 expires 2023-11-17T23:51:04.390538+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3874168431 expires 2023-11-17T23:51:30.957768+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/1477838149 expires 2023-11-17T23:51:04.390538+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/792254178 expires 2023-11-17T23:51:04.390538+0000 2023-11-17T00:01:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1353533979 expires 2023-11-17T23:51:30.957768+0000 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 msgr2=0x7ff1940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 0x7ff1940a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400b2e0 tx=0x7ff1a4005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 msgr2=0x7ff1a8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff19800e9d0 tx=0x7ff19800ece0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 shutdown_connections 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff1940a11e0 0x7ff1940a3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1a81081c0 0x7ff1a819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1a8073550 0x7ff1a8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 --2- 172.21.15.23:0/1172156498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a8074b20 0x7ff1a8198770 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 >> 172.21.15.23:0/1172156498 conn(0x7ff1a80fb900 msgr2=0x7ff1a80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 shutdown_connections 2023-11-17T00:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:01.553+0000 7ff1af75f700 1 -- 172.21.15.23:0/1172156498 wait complete. 2023-11-17T00:01:01.950 INFO:teuthology.run_tasks:Running task fs.ready... 2023-11-17T00:01:01.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 -- 172.21.15.23:0/3743163500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 msgr2=0x7f78281018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 --2- 172.21.15.23:0/3743163500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281018f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7818009990 tx=0x7f7818009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 -- 172.21.15.23:0/3743163500 shutdown_connections 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 --2- 172.21.15.23:0/3743163500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 0x7f78281086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 --2- 172.21.15.23:0/3743163500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f78281006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 --2- 172.21.15.23:0/3743163500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281018f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 -- 172.21.15.23:0/3743163500 >> 172.21.15.23:0/3743163500 conn(0x7f78280fb890 msgr2=0x7f78280fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:02.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 -- 172.21.15.23:0/3743163500 shutdown_connections 2023-11-17T00:01:02.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.185+0000 7f782fbff700 1 -- 172.21.15.23:0/3743163500 wait complete. 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.186+0000 7f782fbff700 1 Processor -- start 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.186+0000 7f782fbff700 1 -- start start 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f7828198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 0x7f782819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782819fd10 con 0x7f78281014a0 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782819fe50 con 0x7f7828101e30 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782fbff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782819ffe0 con 0x7f78281002b0 2023-11-17T00:01:02.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d99b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f7828198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d99b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f7828198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52564/0 (socket says 172.21.15.23:52564) 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d99b700 1 -- 172.21.15.23:0/3009366942 learned_addr learned my addr 172.21.15.23:0/3009366942 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782e19c700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 0x7f782819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 -- 172.21.15.23:0/3009366942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 msgr2=0x7f7828198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f7828198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 -- 172.21.15.23:0/3009366942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 msgr2=0x7f782819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 0x7f782819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.187+0000 7f782d19a700 1 -- 172.21.15.23:0/3009366942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180095f0 con 0x7f78281014a0 2023-11-17T00:01:02.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f782d19a700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281986d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7818000c00 tx=0x7f7818005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781801d070 con 0x7f78281014a0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7818003f60 con 0x7f78281014a0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782810bc60 con 0x7f78281014a0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818022b40 con 0x7f78281014a0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.188+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782810bad0 con 0x7f78281014a0 2023-11-17T00:01:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.189+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78281928f0 con 0x7f78281014a0 2023-11-17T00:01:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.191+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7818003aa0 con 0x7f78281014a0 2023-11-17T00:01:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.192+0000 7f781effd700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 0x7f78140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.192+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f78180ce230 con 0x7f78281014a0 2023-11-17T00:01:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.192+0000 7f782d99b700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 0x7f78140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.193+0000 7f782d99b700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 0x7f78140a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7824009510 tx=0x7f7824009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:02.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.196+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f78180977c0 con 0x7f78281014a0 2023-11-17T00:01:02.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:02 smithi023 ceph-mon[97824]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:01:02.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:02 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1172156498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-17T00:01:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.411+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782802cfe0 con 0x7f78281014a0 2023-11-17T00:01:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.413+0000 7f781effd700 1 -- 172.21.15.23:0/3009366942 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7f781809ade0 con 0x7f78281014a0 2023-11-17T00:01:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:01:02.415 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 msgr2=0x7f78140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 0x7f78140a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7824009510 tx=0x7f7824009040 comp rx=0 tx=0).stop 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 msgr2=0x7f78281986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281986d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7818000c00 tx=0x7f7818005740 comp rx=0 tx=0).stop 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 shutdown_connections 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7828101e30 0x7f782819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78140a1150 0x7f78140a3600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78281002b0 0x7f7828198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 --2- 172.21.15.23:0/3009366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78281014a0 0x7f78281986d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.415+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 >> 172.21.15.23:0/3009366942 conn(0x7f78280fb890 msgr2=0x7f7828106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.416+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 shutdown_connections 2023-11-17T00:01:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.416+0000 7f782fbff700 1 -- 172.21.15.23:0/3009366942 wait complete. 2023-11-17T00:01:02.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:01:02.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:01:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:02 smithi173 ceph-mon[131534]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:01:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:02 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1172156498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-17T00:01:02.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:02 smithi156 ceph-mon[131105]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-11-17T00:01:02.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:02 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1172156498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-17T00:01:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/135264738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4100d40 msgr2=0x7fa6b4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/135264738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4100d40 0x7fa6b4103120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009990 tx=0x7fa6b0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/135264738 shutdown_connections 2023-11-17T00:01:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/135264738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4105f80 0x7fa6b41083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/135264738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4100d40 0x7fa6b4103120 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/135264738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 0x7fa6b4105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/135264738 >> 172.21.15.23:0/135264738 conn(0x7fa6b40fa7c0 msgr2=0x7fa6b40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/135264738 shutdown_connections 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.644+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/135264738 wait complete. 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.646+0000 7fa6bc7ff700 1 Processor -- start 2023-11-17T00:01:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.646+0000 7fa6bc7ff700 1 -- start start 2023-11-17T00:01:02.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.646+0000 7fa6bc7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bc7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 0x7fa6b4196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bc7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4105f80 0x7fa6b419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bc7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b419dc40 con 0x7fa6b4100d40 2023-11-17T00:01:02.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bc7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b419dd80 con 0x7fa6b4103660 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bc7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b419df10 con 0x7fa6b4105f80 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50398/0 (socket says 172.21.15.23:50398) 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 -- 172.21.15.23:0/820475202 learned_addr learned my addr 172.21.15.23:0/820475202 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6bad9c700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4105f80 0x7fa6b419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6b9d9a700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 0x7fa6b4196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 -- 172.21.15.23:0/820475202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4105f80 msgr2=0x7fa6b419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4105f80 0x7fa6b419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 -- 172.21.15.23:0/820475202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 msgr2=0x7fa6b4196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 0x7fa6b4196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.647+0000 7fa6ba59b700 1 -- 172.21.15.23:0/820475202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac009c40 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6ba59b700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b680 tx=0x7fa6b0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b001d070 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6b0021410 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00095f0 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0017650 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.648+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b410bc70 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.649+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b410be00 con 0x7fa6b4100d40 2023-11-17T00:01:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.651+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa6b0021580 con 0x7fa6b4100d40 2023-11-17T00:01:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.652+0000 7fa6ab7fe700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 0x7fa6a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.653+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa6b00cde10 con 0x7fa6b4100d40 2023-11-17T00:01:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.653+0000 7fa6b9d9a700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 0x7fa6a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.653+0000 7fa6b9d9a700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 0x7fa6a00a3680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009c60 tx=0x7fa6a4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.657+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa6b00972e0 con 0x7fa6b4100d40 2023-11-17T00:01:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.870+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6b4061700 con 0x7fa6b4100d40 2023-11-17T00:01:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.871+0000 7fa6ab7fe700 1 -- 172.21.15.23:0/820475202 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6b009a900 con 0x7fa6b4100d40 2023-11-17T00:01:02.872 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:01:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 msgr2=0x7fa6a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 0x7fa6a00a3680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009c60 tx=0x7fa6a4009260 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 msgr2=0x7fa6b41960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b680 tx=0x7fa6b0009580 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 shutdown_connections 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa6a00a11d0 0x7fa6a00a3680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6b4105f80 0x7fa6b419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100d40 0x7fa6b41960c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 --2- 172.21.15.23:0/820475202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6b4103660 0x7fa6b4196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 >> 172.21.15.23:0/820475202 conn(0x7fa6b40fa7c0 msgr2=0x7fa6b41019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 shutdown_connections 2023-11-17T00:01:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:02.873+0000 7fa6bc7ff700 1 -- 172.21.15.23:0/820475202 wait complete. 2023-11-17T00:01:02.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:01:03.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 -- 172.21.15.23:0/1264339968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81002c0 msgr2=0x7f2de81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/1264339968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81002c0 0x7f2de81006d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009990 tx=0x7f2dd8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 -- 172.21.15.23:0/1264339968 shutdown_connections 2023-11-17T00:01:03.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/1264339968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de8101e40 0x7f2de8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/1264339968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81002c0 0x7f2de81006d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/1264339968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de81014b0 0x7f2de8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 -- 172.21.15.23:0/1264339968 >> 172.21.15.23:0/1264339968 conn(0x7f2de80fb8a0 msgr2=0x7f2de80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:03.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 -- 172.21.15.23:0/1264339968 shutdown_connections 2023-11-17T00:01:03.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.100+0000 7f2ded7ff700 1 -- 172.21.15.23:0/1264339968 wait complete. 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.101+0000 7f2ded7ff700 1 Processor -- start 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.101+0000 7f2ded7ff700 1 -- start start 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de81002c0 0x7f2de8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 0x7f2de819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de819fdf0 con 0x7f2de81014b0 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de819ff30 con 0x7f2de81002c0 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ded7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81a00c0 con 0x7f2de8101e40 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ddffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2de77fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 0x7f2de819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ddffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50420/0 (socket says 172.21.15.23:50420) 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2ddffff700 1 -- 172.21.15.23:0/2280260021 learned_addr learned my addr 172.21.15.23:0/2280260021 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.102+0000 7f2de6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de81002c0 0x7f2de8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 -- 172.21.15.23:0/2280260021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 msgr2=0x7f2de819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 0x7f2de819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 -- 172.21.15.23:0/2280260021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de81002c0 msgr2=0x7f2de8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de81002c0 0x7f2de8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 -- 172.21.15.23:0/2280260021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80095f0 con 0x7f2de81014b0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2de77fe700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 0x7f2de819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ddffff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9a0 tx=0x7f2dd000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0009790 con 0x7f2de81014b0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dd0010430 con 0x7f2de81014b0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0018680 con 0x7f2de81014b0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de810bc90 con 0x7f2de81014b0 2023-11-17T00:01:03.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.103+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de810bb00 con 0x7f2de81014b0 2023-11-17T00:01:03.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.104+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de81929f0 con 0x7f2de81014b0 2023-11-17T00:01:03.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.106+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2dd00105a0 con 0x7f2de81014b0 2023-11-17T00:01:03.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.107+0000 7f2de4ff9700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 0x7f2dd40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.108+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2dd0014070 con 0x7f2de81014b0 2023-11-17T00:01:03.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.108+0000 7f2de6ffd700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 0x7f2dd40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.108+0000 7f2de6ffd700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 0x7f2dd40a3680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800b2e0 tx=0x7f2dd8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:03.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.111+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2dd0051730 con 0x7f2de81014b0 2023-11-17T00:01:03.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.325+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2de8061c00 con 0x7f2de81014b0 2023-11-17T00:01:03.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.326+0000 7f2de4ff9700 1 -- 172.21.15.23:0/2280260021 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2dd00961d0 con 0x7f2de81014b0 2023-11-17T00:01:03.328 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 msgr2=0x7f2dd40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 0x7f2dd40a3680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800b2e0 tx=0x7f2dd8005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 msgr2=0x7f2de81987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9a0 tx=0x7f2dd000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 shutdown_connections 2023-11-17T00:01:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f2dd40a11d0 0x7f2dd40a3680 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2de8101e40 0x7f2de819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de81002c0 0x7f2de8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 --2- 172.21.15.23:0/2280260021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2de81014b0 0x7f2de81987b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 >> 172.21.15.23:0/2280260021 conn(0x7f2de80fb8a0 msgr2=0x7f2de8106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 shutdown_connections 2023-11-17T00:01:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.328+0000 7f2ded7ff700 1 -- 172.21.15.23:0/2280260021 wait complete. 2023-11-17T00:01:03.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:01:03.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:03 smithi023 ceph-mon[97824]: mgrmap e18: x(active, since 9m), standbys: z, y 2023-11-17T00:01:03.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:03 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3009366942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:03.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:03 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/820475202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 -- 172.21.15.23:0/2454870753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 msgr2=0x7f0144101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 --2- 172.21.15.23:0/2454870753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144101900 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 -- 172.21.15.23:0/2454870753 shutdown_connections 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 --2- 172.21.15.23:0/2454870753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0144101e40 0x7f0144108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 --2- 172.21.15.23:0/2454870753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01441002c0 0x7f01441006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 --2- 172.21.15.23:0/2454870753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144101900 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 -- 172.21.15.23:0/2454870753 >> 172.21.15.23:0/2454870753 conn(0x7f01440fb8a0 msgr2=0x7f01440fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 -- 172.21.15.23:0/2454870753 shutdown_connections 2023-11-17T00:01:03.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.556+0000 7f014abff700 1 -- 172.21.15.23:0/2454870753 wait complete. 2023-11-17T00:01:03.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.557+0000 7f014abff700 1 Processor -- start 2023-11-17T00:01:03.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 -- start start 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01441002c0 0x7f0144198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0144101e40 0x7f014419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014419fdd0 con 0x7f01441014b0 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014419ff10 con 0x7f01441002c0 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014abff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01441a00a0 con 0x7f0144101e40 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f0143fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f0143fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50424/0 (socket says 172.21.15.23:50424) 2023-11-17T00:01:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f0143fff700 1 -- 172.21.15.23:0/4152479606 learned_addr learned my addr 172.21.15.23:0/4152479606 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014899b700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01441002c0 0x7f0144198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.558+0000 7f014919c700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0144101e40 0x7f014419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 -- 172.21.15.23:0/4152479606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0144101e40 msgr2=0x7f014419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0144101e40 0x7f014419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 -- 172.21.15.23:0/4152479606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01441002c0 msgr2=0x7f0144198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01441002c0 0x7f0144198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 -- 172.21.15.23:0/4152479606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0138009520 con 0x7f01441014b0 2023-11-17T00:01:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0143fff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f013000b620 tx=0x7f0130005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013001d070 con 0x7f01441014b0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0130022470 con 0x7f01441014b0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01300095f0 con 0x7f01441014b0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130017650 con 0x7f01441014b0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.559+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014410bc70 con 0x7f01441014b0 2023-11-17T00:01:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.560+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014410be00 con 0x7f01441014b0 2023-11-17T00:01:03.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.562+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f01300225e0 con 0x7f01441014b0 2023-11-17T00:01:03.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.564+0000 7f0141ffb700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 0x7f01340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:03.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.564+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f01300ce0d0 con 0x7f01441014b0 2023-11-17T00:01:03.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.564+0000 7f014899b700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 0x7f01340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:03.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.565+0000 7f014899b700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 0x7f01340a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0144101310 tx=0x7f0138009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:03.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.567+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f01300975a0 con 0x7f01441014b0 2023-11-17T00:01:03.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:03 smithi173 ceph-mon[131534]: mgrmap e18: x(active, since 9m), standbys: z, y 2023-11-17T00:01:03.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:03 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3009366942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:03.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:03 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/820475202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:03.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:03 smithi156 ceph-mon[131105]: mgrmap e18: x(active, since 9m), standbys: z, y 2023-11-17T00:01:03.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:03 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3009366942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:03.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:03 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/820475202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.776+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0144061700 con 0x7f01441014b0 2023-11-17T00:01:03.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.776+0000 7f0141ffb700 1 -- 172.21.15.23:0/4152479606 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f013009abc0 con 0x7f01441014b0 2023-11-17T00:01:03.778 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 msgr2=0x7f01340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 0x7f01340a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0144101310 tx=0x7f0138009260 comp rx=0 tx=0).stop 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 msgr2=0x7f0144198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f013000b620 tx=0x7f0130005740 comp rx=0 tx=0).stop 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 shutdown_connections 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f01340a1150 0x7f01340a3600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0144101e40 0x7f014419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01441002c0 0x7f0144198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 --2- 172.21.15.23:0/4152479606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01441014b0 0x7f0144198790 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:03.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 >> 172.21.15.23:0/4152479606 conn(0x7f01440fb8a0 msgr2=0x7f0144106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:03.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 shutdown_connections 2023-11-17T00:01:03.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:03.779+0000 7f014abff700 1 -- 172.21.15.23:0/4152479606 wait complete. 2023-11-17T00:01:03.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:01:04.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 -- 172.21.15.23:0/198439134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 msgr2=0x7f22ec103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 --2- 172.21.15.23:0/198439134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec103120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009990 tx=0x7f22e8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 -- 172.21.15.23:0/198439134 shutdown_connections 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 --2- 172.21.15.23:0/198439134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec105f80 0x7f22ec1083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 --2- 172.21.15.23:0/198439134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec103120 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 --2- 172.21.15.23:0/198439134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec103660 0x7f22ec105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.006+0000 7f22f35ff700 1 -- 172.21.15.23:0/198439134 >> 172.21.15.23:0/198439134 conn(0x7f22ec0fa7c0 msgr2=0x7f22ec0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.007+0000 7f22f35ff700 1 -- 172.21.15.23:0/198439134 shutdown_connections 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.007+0000 7f22f35ff700 1 -- 172.21.15.23:0/198439134 wait complete. 2023-11-17T00:01:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.008+0000 7f22f35ff700 1 Processor -- start 2023-11-17T00:01:04.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.008+0000 7f22f35ff700 1 -- start start 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec1006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec105f80 0x7f22ec0ff230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0ff9d0 con 0x7f22ec103660 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0ffb40 con 0x7f22ec105f80 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f35ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec1a00a0 con 0x7f22ec100d40 2023-11-17T00:01:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50446/0 (socket says 172.21.15.23:50446) 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 -- 172.21.15.23:0/1010422902 learned_addr learned my addr 172.21.15.23:0/1010422902 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f139b700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec1006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f1b9c700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec105f80 0x7f22ec0ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 -- 172.21.15.23:0/1010422902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 msgr2=0x7f22ec1006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec1006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 -- 172.21.15.23:0/1010422902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec105f80 msgr2=0x7f22ec0ff230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec105f80 0x7f22ec0ff230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.009+0000 7f22f0b9a700 1 -- 172.21.15.23:0/1010422902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e80095f0 con 0x7f22ec103660 2023-11-17T00:01:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22f0b9a700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9a0 tx=0x7f22dc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc009790 con 0x7f22ec103660 2023-11-17T00:01:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec10bc50 con 0x7f22ec103660 2023-11-17T00:01:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22dc010430 con 0x7f22ec103660 2023-11-17T00:01:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc018680 con 0x7f22ec103660 2023-11-17T00:01:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.010+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec10bac0 con 0x7f22ec103660 2023-11-17T00:01:04.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.011+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec0fc330 con 0x7f22ec103660 2023-11-17T00:01:04.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.013+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f22dc0105a0 con 0x7f22ec103660 2023-11-17T00:01:04.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.014+0000 7f22e27fc700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 0x7f22d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.015+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f22dc014070 con 0x7f22ec103660 2023-11-17T00:01:04.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.015+0000 7f22f139b700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 0x7f22d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.015+0000 7f22f139b700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 0x7f22d80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b2e0 tx=0x7f22e800b610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.017+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f22dc096410 con 0x7f22ec103660 2023-11-17T00:01:04.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.229+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f22ec061700 con 0x7f22ec103660 2023-11-17T00:01:04.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.230+0000 7f22e27fc700 1 -- 172.21.15.23:0/1010422902 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f22dc0260a0 con 0x7f22ec103660 2023-11-17T00:01:04.231 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:01:04.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 msgr2=0x7f22d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 0x7f22d80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b2e0 tx=0x7f22e800b610 comp rx=0 tx=0).stop 2023-11-17T00:01:04.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 msgr2=0x7f22ec0fecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9a0 tx=0x7f22dc00ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 shutdown_connections 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22ec105f80 0x7f22ec0ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f22d80a1150 0x7f22d80a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22ec100d40 0x7f22ec1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 --2- 172.21.15.23:0/1010422902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22ec103660 0x7f22ec0fecf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.232+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 >> 172.21.15.23:0/1010422902 conn(0x7f22ec0fa7c0 msgr2=0x7f22ec1019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.233+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 shutdown_connections 2023-11-17T00:01:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.233+0000 7f22f35ff700 1 -- 172.21.15.23:0/1010422902 wait complete. 2023-11-17T00:01:04.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:01:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 -- 172.21.15.23:0/3255369855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0101e10 msgr2=0x7f65b01086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 --2- 172.21.15.23:0/3255369855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0101e10 0x7f65b01086d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65a800b0a0 tx=0x7f65a800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 -- 172.21.15.23:0/3255369855 shutdown_connections 2023-11-17T00:01:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 --2- 172.21.15.23:0/3255369855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0101e10 0x7f65b01086d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 --2- 172.21.15.23:0/3255369855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0100290 0x7f65b01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 --2- 172.21.15.23:0/3255369855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101480 0x7f65b01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 -- 172.21.15.23:0/3255369855 >> 172.21.15.23:0/3255369855 conn(0x7f65b00fb870 msgr2=0x7f65b00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 -- 172.21.15.23:0/3255369855 shutdown_connections 2023-11-17T00:01:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.455+0000 7f65b77ff700 1 -- 172.21.15.23:0/3255369855 wait complete. 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.456+0000 7f65b77ff700 1 Processor -- start 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.456+0000 7f65b77ff700 1 -- start start 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0101480 0x7f65b01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101e10 0x7f65b019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b019fe10 con 0x7f65b0100290 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b019ff50 con 0x7f65b0101e10 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b77ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01a00e0 con 0x7f65b0101480 2023-11-17T00:01:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b559b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b559b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50454/0 (socket says 172.21.15.23:50454) 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101e10 0x7f65b019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b559b700 1 -- 172.21.15.23:0/1165577419 learned_addr learned my addr 172.21.15.23:0/1165577419 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.457+0000 7f65b4d9a700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0101480 0x7f65b01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 -- 172.21.15.23:0/1165577419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0101480 msgr2=0x7f65b01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0101480 0x7f65b01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 -- 172.21.15.23:0/1165577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101e10 msgr2=0x7f65b019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101e10 0x7f65b019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 -- 172.21.15.23:0/1165577419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8009d00 con 0x7f65b0100290 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b559b700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00e970 tx=0x7f65ac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac009790 con 0x7f65b0100290 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65ac00c7e0 con 0x7f65b0100290 2023-11-17T00:01:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b010bc70 con 0x7f65b0100290 2023-11-17T00:01:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac003c70 con 0x7f65b0100290 2023-11-17T00:01:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.458+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b010bae0 con 0x7f65b0100290 2023-11-17T00:01:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.459+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b01a0830 con 0x7f65b0100290 2023-11-17T00:01:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.461+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f65ac00cc50 con 0x7f65b0100290 2023-11-17T00:01:04.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.462+0000 7f65a67fc700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 0x7f659c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.463+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f65ac014070 con 0x7f65b0100290 2023-11-17T00:01:04.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.463+0000 7f65b4d9a700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 0x7f659c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.463+0000 7f65b4d9a700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 0x7f659c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009c60 tx=0x7f65a0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.466+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f65ac096210 con 0x7f65b0100290 2023-11-17T00:01:04.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:04 smithi023 ceph-mon[97824]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-11-17T00:01:04.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:04 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2280260021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:04 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4152479606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:04 smithi173 ceph-mon[131534]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:04 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2280260021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:04 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4152479606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:04 smithi156 ceph-mon[131105]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:04 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2280260021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:04 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4152479606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:04.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.722+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b01a0de0 con 0x7f65b0100290 2023-11-17T00:01:04.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.723+0000 7f65a67fc700 1 -- 172.21.15.23:0/1165577419 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7f65ac003dd0 con 0x7f65b0100290 2023-11-17T00:01:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:01:04.725 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:01:04.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 msgr2=0x7f659c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 0x7f659c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009c60 tx=0x7f65a0009260 comp rx=0 tx=0).stop 2023-11-17T00:01:04.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 msgr2=0x7f65b0198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00e970 tx=0x7f65ac00ec80 comp rx=0 tx=0).stop 2023-11-17T00:01:04.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 shutdown_connections 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65b0101e10 0x7f65b019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b0100290 0x7f65b0198290 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f659c0a1150 0x7f659c0a3600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 --2- 172.21.15.23:0/1165577419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65b0101480 0x7f65b01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 >> 172.21.15.23:0/1165577419 conn(0x7f65b00fb870 msgr2=0x7f65b00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 shutdown_connections 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.726+0000 7f65b77ff700 1 -- 172.21.15.23:0/1165577419 wait complete. 2023-11-17T00:01:04.728 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:01:04.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:01:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 -- 172.21.15.23:0/3303201046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878073550 msgr2=0x7f78780739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 --2- 172.21.15.23:0/3303201046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878073550 0x7f78780739c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f786c009990 tx=0x7f786c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 -- 172.21.15.23:0/3303201046 shutdown_connections 2023-11-17T00:01:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 --2- 172.21.15.23:0/3303201046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f787810c860 0x7f787810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 --2- 172.21.15.23:0/3303201046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7878074b30 0x7f7878072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 --2- 172.21.15.23:0/3303201046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878073550 0x7f78780739c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 -- 172.21.15.23:0/3303201046 >> 172.21.15.23:0/3303201046 conn(0x7f78780fb870 msgr2=0x7f78780fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 -- 172.21.15.23:0/3303201046 shutdown_connections 2023-11-17T00:01:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.951+0000 7f787d5ff700 1 -- 172.21.15.23:0/3303201046 wait complete. 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.952+0000 7f787d5ff700 1 Processor -- start 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 -- start start 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f787810c860 0x7f78781a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78781a4170 con 0x7f7878074b30 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78781a42b0 con 0x7f787810c860 2023-11-17T00:01:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f787d5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78781a4440 con 0x7f7878073550 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f78767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f7876ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f78777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f787810c860 0x7f78781a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f7876ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52656/0 (socket says 172.21.15.23:52656) 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f7876ffd700 1 -- 172.21.15.23:0/354414352 learned_addr learned my addr 172.21.15.23:0/354414352 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.953+0000 7f78767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50482/0 (socket says 172.21.15.23:50482) 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 -- 172.21.15.23:0/354414352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f787810c860 msgr2=0x7f78781a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f787810c860 0x7f78781a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 -- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 msgr2=0x7f787819cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 -- 172.21.15.23:0/354414352 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0095f0 con 0x7f7878073550 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f78767fc700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f7876ffd700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f786800d980 tx=0x7f786800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786800f890 con 0x7f7878073550 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7868010460 con 0x7f7878073550 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7878112270 con 0x7f7878073550 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786800e610 con 0x7f7878073550 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.954+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78781120e0 con 0x7f7878073550 2023-11-17T00:01:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.955+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78781023b0 con 0x7f7878073550 2023-11-17T00:01:04.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.957+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f78680105d0 con 0x7f7878073550 2023-11-17T00:01:04.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.959+0000 7f785ffff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 0x7f78600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:04.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.959+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f78680cc670 con 0x7f7878073550 2023-11-17T00:01:04.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.959+0000 7f78767fc700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 0x7f78600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:04.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.959+0000 7f78767fc700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 0x7f78600a3600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f786c00b5c0 tx=0x7f786c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:04.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:04.963+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7868095a50 con 0x7f7878073550 2023-11-17T00:01:05.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.178+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f787802cfe0 con 0x7f7878073550 2023-11-17T00:01:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.188+0000 7f785ffff700 1 -- 172.21.15.23:0/354414352 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7f7868099070 con 0x7f7878073550 2023-11-17T00:01:05.190 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:01:05.190 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 msgr2=0x7f78600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 0x7f78600a3600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f786c00b5c0 tx=0x7f786c005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 msgr2=0x7f787819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f786800d980 tx=0x7f786800dc90 comp rx=0 tx=0).stop 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 shutdown_connections 2023-11-17T00:01:05.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f787810c860 0x7f78781a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f78600a1150 0x7f78600a3600 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7878073550 0x7f787819c5f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 --2- 172.21.15.23:0/354414352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7878074b30 0x7f787819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.190+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 >> 172.21.15.23:0/354414352 conn(0x7f78780fb870 msgr2=0x7f78780fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.191+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 shutdown_connections 2023-11-17T00:01:05.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.191+0000 7f787d5ff700 1 -- 172.21.15.23:0/354414352 wait complete. 2023-11-17T00:01:05.212 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:01:05.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 -- 172.21.15.23:0/1719037253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801002c0 msgr2=0x7ff5801006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 --2- 172.21.15.23:0/1719037253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801002c0 0x7ff5801006d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff570009990 tx=0x7ff570009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 -- 172.21.15.23:0/1719037253 shutdown_connections 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 --2- 172.21.15.23:0/1719037253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 0x7ff580108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 --2- 172.21.15.23:0/1719037253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801002c0 0x7ff5801006d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 --2- 172.21.15.23:0/1719037253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801014b0 0x7ff580101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 -- 172.21.15.23:0/1719037253 >> 172.21.15.23:0/1719037253 conn(0x7ff5800fb8a0 msgr2=0x7ff5800fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 -- 172.21.15.23:0/1719037253 shutdown_connections 2023-11-17T00:01:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.418+0000 7ff5851ff700 1 -- 172.21.15.23:0/1719037253 wait complete. 2023-11-17T00:01:05.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 Processor -- start 2023-11-17T00:01:05.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 -- start start 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801002c0 0x7ff5800748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 0x7ff580073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff580073bc0 con 0x7ff5801014b0 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff580073d30 con 0x7ff5801002c0 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.419+0000 7ff5851ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5801a00f0 con 0x7ff580101e40 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50500/0 (socket says 172.21.15.23:50500) 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 -- 172.21.15.23:0/4095507309 learned_addr learned my addr 172.21.15.23:0/4095507309 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57ed9d700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801002c0 0x7ff5800748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57f59e700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 0x7ff580073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 -- 172.21.15.23:0/4095507309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 msgr2=0x7ff580073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 0x7ff580073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 -- 172.21.15.23:0/4095507309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801002c0 msgr2=0x7ff5800748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801002c0 0x7ff5800748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 -- 172.21.15.23:0/4095507309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5700095f0 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.420+0000 7ff57e59c700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff57400e9a0 tx=0x7ff57400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.421+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574009790 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.421+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff574010430 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.421+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58010bc90 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.421+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574018680 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.421+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58010bb00 con 0x7ff5801014b0 2023-11-17T00:01:05.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.422+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff580107850 con 0x7ff5801014b0 2023-11-17T00:01:05.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.424+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5740105a0 con 0x7ff5801014b0 2023-11-17T00:01:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.425+0000 7ff567fff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 0x7ff5680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.425+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff574014070 con 0x7ff5801014b0 2023-11-17T00:01:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.425+0000 7ff57ed9d700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 0x7ff5680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.426+0000 7ff57ed9d700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 0x7ff5680a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff57000b2e0 tx=0x7ff570005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.429+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff574051730 con 0x7ff5801014b0 2023-11-17T00:01:05.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:05 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1010422902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:05.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:05 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1165577419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:05.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:05 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1010422902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:05.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:05 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1165577419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:05.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:05 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1010422902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:05.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:05 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1165577419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:05.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.636+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff580061bc0 con 0x7ff5801014b0 2023-11-17T00:01:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.636+0000 7ff567fff700 1 -- 172.21.15.23:0/4095507309 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff5740961d0 con 0x7ff5801014b0 2023-11-17T00:01:05.638 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:01:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.638+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 msgr2=0x7ff5680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.638+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 0x7ff5680a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff57000b2e0 tx=0x7ff570005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.638+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 msgr2=0x7ff580072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.638+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff57400e9a0 tx=0x7ff57400ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 shutdown_connections 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff5680a11d0 0x7ff5680a3680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff580101e40 0x7ff580073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5801002c0 0x7ff5800748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 --2- 172.21.15.23:0/4095507309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5801014b0 0x7ff580072ee0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 >> 172.21.15.23:0/4095507309 conn(0x7ff5800fb8a0 msgr2=0x7ff580104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 shutdown_connections 2023-11-17T00:01:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.639+0000 7ff5851ff700 1 -- 172.21.15.23:0/4095507309 wait complete. 2023-11-17T00:01:05.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:01:05.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 -- 172.21.15.23:0/55238448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41002c0 msgr2=0x7fe3e41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/55238448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41002c0 0x7fe3e41006d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009990 tx=0x7fe3e0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 -- 172.21.15.23:0/55238448 shutdown_connections 2023-11-17T00:01:05.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/55238448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 0x7fe3e4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/55238448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41002c0 0x7fe3e41006d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/55238448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41014b0 0x7fe3e4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 -- 172.21.15.23:0/55238448 >> 172.21.15.23:0/55238448 conn(0x7fe3e40fb8a0 msgr2=0x7fe3e40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 -- 172.21.15.23:0/55238448 shutdown_connections 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.866+0000 7fe3ebdff700 1 -- 172.21.15.23:0/55238448 wait complete. 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.867+0000 7fe3ebdff700 1 Processor -- start 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.867+0000 7fe3ebdff700 1 -- start start 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41002c0 0x7fe3e4196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 0x7fe3e419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e419dc10 con 0x7fe3e41014b0 2023-11-17T00:01:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e419dd50 con 0x7fe3e41002c0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ebdff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e419dee0 con 0x7fe3e4101e40 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50528/0 (socket says 172.21.15.23:50528) 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e939a700 1 -- 172.21.15.23:0/3693927607 learned_addr learned my addr 172.21.15.23:0/3693927607 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3ea39c700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 0x7fe3e419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e9b9b700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41002c0 0x7fe3e4196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e939a700 1 -- 172.21.15.23:0/3693927607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 msgr2=0x7fe3e419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.868+0000 7fe3e939a700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 0x7fe3e419d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3e939a700 1 -- 172.21.15.23:0/3693927607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41002c0 msgr2=0x7fe3e4196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3e939a700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41002c0 0x7fe3e4196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3e939a700 1 -- 172.21.15.23:0/3693927607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e00095f0 con 0x7fe3e41014b0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3e939a700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400e9a0 tx=0x7fe3d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4009790 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3d4010430 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e410bc90 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4018680 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.869+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e410bb00 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.870+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e4107850 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.872+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe3d40105a0 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.874+0000 7fe3daffd700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 0x7fe3d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.874+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe3d4014070 con 0x7fe3e41014b0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.874+0000 7fe3e9b9b700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 0x7fe3d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:05.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.875+0000 7fe3e9b9b700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 0x7fe3d00a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000b2e0 tx=0x7fe3e0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:05.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:05.877+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe3d4051730 con 0x7fe3e41014b0 2023-11-17T00:01:06.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.091+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe3e4061950 con 0x7fe3e41014b0 2023-11-17T00:01:06.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.092+0000 7fe3daffd700 1 -- 172.21.15.23:0/3693927607 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe3d4096220 con 0x7fe3e41014b0 2023-11-17T00:01:06.093 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 msgr2=0x7fe3d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 0x7fe3d00a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000b2e0 tx=0x7fe3e0005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 msgr2=0x7fe3e41965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400e9a0 tx=0x7fe3d400ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 shutdown_connections 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d00a1150 0x7fe3d00a3600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3e4101e40 0x7fe3e419d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3e41002c0 0x7fe3e4196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 --2- 172.21.15.23:0/3693927607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3e41014b0 0x7fe3e41965d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 >> 172.21.15.23:0/3693927607 conn(0x7fe3e40fb8a0 msgr2=0x7fe3e4104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:06.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 shutdown_connections 2023-11-17T00:01:06.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.094+0000 7fe3ebdff700 1 -- 172.21.15.23:0/3693927607 wait complete. 2023-11-17T00:01:06.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:01:06.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 -- 172.21.15.23:0/2582388507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 msgr2=0x7f88681006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2582388507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88681006e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8858009990 tx=0x7f8858009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 -- 172.21.15.23:0/2582388507 shutdown_connections 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2582388507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8868101e50 0x7f8868108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2582388507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88681006e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2582388507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681014c0 0x7f8868101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 -- 172.21.15.23:0/2582388507 >> 172.21.15.23:0/2582388507 conn(0x7f88680fb8b0 msgr2=0x7f88680fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:06.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 -- 172.21.15.23:0/2582388507 shutdown_connections 2023-11-17T00:01:06.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.326+0000 7f886c9ff700 1 -- 172.21.15.23:0/2582388507 wait complete. 2023-11-17T00:01:06.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.327+0000 7f886c9ff700 1 Processor -- start 2023-11-17T00:01:06.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.327+0000 7f886c9ff700 1 -- start start 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.327+0000 7f886c9ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886c9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 0x7f8868072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886c9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 0x7f8868073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886c9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868073bc0 con 0x7f88681014c0 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886c9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868073d30 con 0x7f8868101e50 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886c9ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88681a00d0 con 0x7f88681002d0 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52704/0 (socket says 172.21.15.23:52704) 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 -- 172.21.15.23:0/2335132854 learned_addr learned my addr 172.21.15.23:0/2335132854 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f8866d9d700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 0x7f8868073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f8865d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 0x7f8868072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 -- 172.21.15.23:0/2335132854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 msgr2=0x7f8868073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 0x7f8868073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 -- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 msgr2=0x7f8868072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 0x7f8868072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f886659c700 1 -- 172.21.15.23:0/2335132854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c009520 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f8866d9d700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 0x7f8868073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.328+0000 7f8865d9b700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 0x7f8868072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f886659c700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8858005720 tx=0x7f8858009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885801d070 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8858021410 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580095f0 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858017650 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.329+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886810bc20 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.330+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8868067b00 con 0x7f88681002d0 2023-11-17T00:01:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.332+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8858021580 con 0x7f88681002d0 2023-11-17T00:01:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.333+0000 7f88577fe700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 0x7f88500a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.333+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88580cdfc0 con 0x7f88681002d0 2023-11-17T00:01:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.333+0000 7f8865d9b700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 0x7f88500a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.334+0000 7f8865d9b700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 0x7f88500a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f885c009fd0 tx=0x7f885c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.335+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8858097430 con 0x7f88681002d0 2023-11-17T00:01:06.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:06 smithi023 ceph-mon[97824]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:06 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/354414352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:06 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4095507309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:06 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:06 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3693927607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.557+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f88681a0c70 con 0x7f88681002d0 2023-11-17T00:01:06.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.558+0000 7f88577fe700 1 -- 172.21.15.23:0/2335132854 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8858026020 con 0x7f88681002d0 2023-11-17T00:01:06.559 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 msgr2=0x7f88500a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 0x7f88500a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f885c009fd0 tx=0x7f885c009260 comp rx=0 tx=0).stop 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 msgr2=0x7f88680748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8858005720 tx=0x7f8858009580 comp rx=0 tx=0).stop 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 shutdown_connections 2023-11-17T00:01:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e50 0x7f8868073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a11d0 0x7f88500a3680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681002d0 0x7f88680748a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 --2- 172.21.15.23:0/2335132854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014c0 0x7f8868072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 >> 172.21.15.23:0/2335132854 conn(0x7f88680fb8b0 msgr2=0x7f88680fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 shutdown_connections 2023-11-17T00:01:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.560+0000 7f886c9ff700 1 -- 172.21.15.23:0/2335132854 wait complete. 2023-11-17T00:01:06.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:01:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:06 smithi173 ceph-mon[131534]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:06 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/354414352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:06 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4095507309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:06 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:06 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3693927607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:06 smithi156 ceph-mon[131105]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:06.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:06 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/354414352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:06 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4095507309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:06 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:01:06.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:06 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3693927607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:01:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 -- 172.21.15.23:0/1184653317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8103660 msgr2=0x7f8ef8105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1184653317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8103660 0x7f8ef8105a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009990 tx=0x7f8ee8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 -- 172.21.15.23:0/1184653317 shutdown_connections 2023-11-17T00:01:06.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1184653317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8105f80 0x7f8ef81083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1184653317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 0x7f8ef8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1184653317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8103660 0x7f8ef8105a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 -- 172.21.15.23:0/1184653317 >> 172.21.15.23:0/1184653317 conn(0x7f8ef80fa7c0 msgr2=0x7f8ef80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 -- 172.21.15.23:0/1184653317 shutdown_connections 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.801+0000 7f8f003ff700 1 -- 172.21.15.23:0/1184653317 wait complete. 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 Processor -- start 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 -- start start 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 0x7f8ef8196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8105f80 0x7f8ef819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef819dc00 con 0x7f8ef8103660 2023-11-17T00:01:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef819dd40 con 0x7f8ef8100d40 2023-11-17T00:01:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.802+0000 7f8f003ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef819ded0 con 0x7f8ef8105f80 2023-11-17T00:01:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efe99c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8105f80 0x7f8ef819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50556/0 (socket says 172.21.15.23:50556) 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 -- 172.21.15.23:0/1420733077 learned_addr learned my addr 172.21.15.23:0/1420733077 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efe19b700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 0x7f8ef8196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 -- 172.21.15.23:0/1420733077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8105f80 msgr2=0x7f8ef819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8105f80 0x7f8ef819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 -- 172.21.15.23:0/1420733077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 msgr2=0x7f8ef8196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 0x7f8ef8196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 -- 172.21.15.23:0/1420733077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80095f0 con 0x7f8ef8103660 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8efd99a700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80092e0 tx=0x7f8ee8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.803+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee801d070 con 0x7f8ef8103660 2023-11-17T00:01:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.804+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ee8005030 con 0x7f8ef8103660 2023-11-17T00:01:06.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.804+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef810bcd0 con 0x7f8ef8103660 2023-11-17T00:01:06.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.804+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8004b90 con 0x7f8ef8103660 2023-11-17T00:01:06.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.804+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef810bb40 con 0x7f8ef8103660 2023-11-17T00:01:06.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.805+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef80fc330 con 0x7f8ef8103660 2023-11-17T00:01:06.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.807+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ee80051a0 con 0x7f8ef8103660 2023-11-17T00:01:06.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.809+0000 7f8eef7fe700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 0x7f8ee40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:06.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.809+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ee80cdfe0 con 0x7f8ef8103660 2023-11-17T00:01:06.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.809+0000 7f8efe19b700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 0x7f8ee40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:06.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.810+0000 7f8efe19b700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 0x7f8ee40a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009c60 tx=0x7f8ef4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:06.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:06.811+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ee8097450 con 0x7f8ef8103660 2023-11-17T00:01:07.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.031+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8ef8067b00 con 0x7f8ef8103660 2023-11-17T00:01:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.032+0000 7f8eef7fe700 1 -- 172.21.15.23:0/1420733077 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8ee8026020 con 0x7f8ef8103660 2023-11-17T00:01:07.034 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:01:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 msgr2=0x7f8ee40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 0x7f8ee40a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009c60 tx=0x7f8ef4009260 comp rx=0 tx=0).stop 2023-11-17T00:01:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 msgr2=0x7f8ef81965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80092e0 tx=0x7f8ee8009580 comp rx=0 tx=0).stop 2023-11-17T00:01:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 shutdown_connections 2023-11-17T00:01:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8ee40a1150 0x7f8ee40a3600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ef8105f80 0x7f8ef819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ef8100d40 0x7f8ef8196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 --2- 172.21.15.23:0/1420733077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ef8103660 0x7f8ef81965c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 >> 172.21.15.23:0/1420733077 conn(0x7f8ef80fa7c0 msgr2=0x7f8ef8101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.034+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 shutdown_connections 2023-11-17T00:01:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.035+0000 7f8f003ff700 1 -- 172.21.15.23:0/1420733077 wait complete. 2023-11-17T00:01:07.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:01:07.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 -- 172.21.15.23:0/85248451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 msgr2=0x7f87b01006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 --2- 172.21.15.23:0/85248451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b01006e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009990 tx=0x7f87a0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 -- 172.21.15.23:0/85248451 shutdown_connections 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 --2- 172.21.15.23:0/85248451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 0x7f87b01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 --2- 172.21.15.23:0/85248451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b01006e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 --2- 172.21.15.23:0/85248451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 -- 172.21.15.23:0/85248451 >> 172.21.15.23:0/85248451 conn(0x7f87b00fb930 msgr2=0x7f87b00fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.269+0000 7f87b4dff700 1 -- 172.21.15.23:0/85248451 shutdown_connections 2023-11-17T00:01:07.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.270+0000 7f87b4dff700 1 -- 172.21.15.23:0/85248451 wait complete. 2023-11-17T00:01:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.270+0000 7f87b4dff700 1 Processor -- start 2023-11-17T00:01:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 -- start start 2023-11-17T00:01:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b0195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 0x7f87b019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b019db30 con 0x7f87b01002d0 2023-11-17T00:01:07.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b019dc70 con 0x7f87b0101560 2023-11-17T00:01:07.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87b4dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b019de00 con 0x7f87b0101f10 2023-11-17T00:01:07.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87add9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87ae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b0195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87add9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57450/0 (socket says 172.21.15.23:57450) 2023-11-17T00:01:07.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87add9b700 1 -- 172.21.15.23:0/429860594 learned_addr learned my addr 172.21.15.23:0/429860594 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.271+0000 7f87aed9d700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 0x7f87b019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 -- 172.21.15.23:0/429860594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 msgr2=0x7f87b019d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 0x7f87b019d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 -- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 msgr2=0x7f87b0195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b0195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 -- 172.21.15.23:0/429860594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a00095f0 con 0x7f87b0101560 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87add9b700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87a400e9a0 tx=0x7f87a400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4009790 con 0x7f87b0101560 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87a4010430 con 0x7f87b0101560 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b010bcc0 con 0x7f87b0101560 2023-11-17T00:01:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.272+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4018680 con 0x7f87b0101560 2023-11-17T00:01:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.273+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b010bb30 con 0x7f87b0101560 2023-11-17T00:01:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.274+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0107900 con 0x7f87b0101560 2023-11-17T00:01:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.275+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f87a400ca10 con 0x7f87b0101560 2023-11-17T00:01:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.277+0000 7f879f7fe700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 0x7f87980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.277+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f87a4014070 con 0x7f87b0101560 2023-11-17T00:01:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.277+0000 7f87ae59c700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 0x7f87980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.278+0000 7f87ae59c700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 0x7f87980a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b2e0 tx=0x7f87a0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.281+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f87a4051730 con 0x7f87b0101560 2023-11-17T00:01:07.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:07 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2335132854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:07 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1420733077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.545+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b002cfe0 con 0x7f87b0101560 2023-11-17T00:01:07.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.546+0000 7f879f7fe700 1 -- 172.21.15.23:0/429860594 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10814 (secure 0 0 0) 0x7f87a4096230 con 0x7f87b0101560 2023-11-17T00:01:07.548 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:01:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1743311827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1743311827},{"type":"v1","addr":"172.21.15.173:6833","nonce":1743311827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-16T23:59:34.708319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1936617634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1936617634},{"type":"v1","addr":"172.21.15.173:6837","nonce":1936617634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:01:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.548+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 msgr2=0x7f87980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.548+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 0x7f87980a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b2e0 tx=0x7f87a0005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.548+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 msgr2=0x7f87b01964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.548+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87a400e9a0 tx=0x7f87a400ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 shutdown_connections 2023-11-17T00:01:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f87980a1150 0x7f87980a3600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87b0101f10 0x7f87b019d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87b01002d0 0x7f87b0195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 --2- 172.21.15.23:0/429860594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b0101560 0x7f87b01964f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 >> 172.21.15.23:0/429860594 conn(0x7f87b00fb930 msgr2=0x7f87b0104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 shutdown_connections 2023-11-17T00:01:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.549+0000 7f87b4dff700 1 -- 172.21.15.23:0/429860594 wait complete. 2023-11-17T00:01:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2023-11-17T00:01:07.570 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:01:07.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:07 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2335132854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:07 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1420733077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:07 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2335132854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:07 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1420733077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:01:07.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 -- 172.21.15.23:0/2249350686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101ea0 msgr2=0x7fe220108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 --2- 172.21.15.23:0/2249350686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101ea0 0x7fe220108760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe21800b0a0 tx=0x7fe21800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 -- 172.21.15.23:0/2249350686 shutdown_connections 2023-11-17T00:01:07.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 --2- 172.21.15.23:0/2249350686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101ea0 0x7fe220108760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 --2- 172.21.15.23:0/2249350686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 0x7fe220100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 --2- 172.21.15.23:0/2249350686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101510 0x7fe220101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 -- 172.21.15.23:0/2249350686 >> 172.21.15.23:0/2249350686 conn(0x7fe2200fb900 msgr2=0x7fe2200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:07.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 -- 172.21.15.23:0/2249350686 shutdown_connections 2023-11-17T00:01:07.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.816+0000 7fe225dff700 1 -- 172.21.15.23:0/2249350686 wait complete. 2023-11-17T00:01:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.818+0000 7fe225dff700 1 Processor -- start 2023-11-17T00:01:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.818+0000 7fe225dff700 1 -- start start 2023-11-17T00:01:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.818+0000 7fe225dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 0x7fe220198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe225dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe225dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 0x7fe22019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe225dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22019fcc0 con 0x7fe220101ea0 2023-11-17T00:01:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe225dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22019fe00 con 0x7fe220100320 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe225dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22019ff90 con 0x7fe220101510 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52736/0 (socket says 172.21.15.23:52736) 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 -- 172.21.15.23:0/913429548 learned_addr learned my addr 172.21.15.23:0/913429548 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21f7fe700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 0x7fe220198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21ffff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 0x7fe22019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 -- 172.21.15.23:0/913429548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 msgr2=0x7fe220198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 0x7fe220198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 -- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 msgr2=0x7fe22019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 0x7fe22019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21effd700 1 -- 172.21.15.23:0/913429548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe214009520 con 0x7fe220101510 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.819+0000 7fe21ffff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 0x7fe22019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe21effd700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe21400eb30 tx=0x7fe21400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214009880 con 0x7fe220101510 2023-11-17T00:01:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2140044e0 con 0x7fe220101510 2023-11-17T00:01:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe218009d00 con 0x7fe220101510 2023-11-17T00:01:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214003e90 con 0x7fe220101510 2023-11-17T00:01:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.820+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22010bc50 con 0x7fe220101510 2023-11-17T00:01:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.821+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe220067eb0 con 0x7fe220101510 2023-11-17T00:01:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.824+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe214004000 con 0x7fe220101510 2023-11-17T00:01:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.826+0000 7fe21cff9700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 0x7fe2080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.826+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe214014070 con 0x7fe220101510 2023-11-17T00:01:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.826+0000 7fe21f7fe700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 0x7fe2080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.827+0000 7fe21f7fe700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 0x7fe2080a3680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe220101370 tx=0x7fe210009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:07.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:07.830+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe214096490 con 0x7fe220101510 2023-11-17T00:01:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.073+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2201a0bd0 con 0x7fe220101510 2023-11-17T00:01:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.075+0000 7fe21cff9700 1 -- 172.21.15.23:0/913429548 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24510 (secure 0 0 0) 0x7fe214099ab0 con 0x7fe220101510 2023-11-17T00:01:08.078 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:01:08.079 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:00:32.047239+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:01:08.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.077+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 msgr2=0x7fe2080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.077+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 0x7fe2080a3680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe220101370 tx=0x7fe210009190 comp rx=0 tx=0).stop 2023-11-17T00:01:08.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.077+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 msgr2=0x7fe220198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.077+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe21400eb30 tx=0x7fe21400ee40 comp rx=0 tx=0).stop 2023-11-17T00:01:08.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 shutdown_connections 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe220101ea0 0x7fe22019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe220100320 0x7fe220198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe2080a11d0 0x7fe2080a3680 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 --2- 172.21.15.23:0/913429548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe220101510 0x7fe220198680 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 >> 172.21.15.23:0/913429548 conn(0x7fe2200fb900 msgr2=0x7fe2200fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 shutdown_connections 2023-11-17T00:01:08.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.078+0000 7fe225dff700 1 -- 172.21.15.23:0/913429548 wait complete. 2023-11-17T00:01:08.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T23:58:46.253225+0000', 'modified': '2023-11-16T23:59:34.708319+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24572, 'mds_1': 24565, 'mds_2': 24529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/3543343885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 3543343885}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 3543343885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24529': {'gid': 24529, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.173:6835/405519546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 405519546}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 405519546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/180108175', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 180108175}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 180108175}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24541': {'gid': 24541, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6837/1936617634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1936617634}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1936617634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24565': {'gid': 24565, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.173:6839/3306732798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 3306732798}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 3306732798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6839/1637342449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1637342449}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1637342449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-17T00:01:08.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-17T00:01:08.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-11-17T00:01:08.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 -- 172.21.15.23:0/4026422634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444074b30 msgr2=0x7fa444072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/4026422634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444074b30 0x7fa444072f80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa440009990 tx=0x7fa440009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 -- 172.21.15.23:0/4026422634 shutdown_connections 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/4026422634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 0x7fa44410a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/4026422634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444074b30 0x7fa444072f80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/4026422634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444073550 0x7fa4440739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 -- 172.21.15.23:0/4026422634 >> 172.21.15.23:0/4026422634 conn(0x7fa4440fb810 msgr2=0x7fa4440fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 -- 172.21.15.23:0/4026422634 shutdown_connections 2023-11-17T00:01:08.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.316+0000 7fa44b7ff700 1 -- 172.21.15.23:0/4026422634 wait complete. 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 Processor -- start 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 -- start start 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444074b30 0x7fa444198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 0x7fa44419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44419fdc0 con 0x7fa444073550 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44419ff00 con 0x7fa444074b30 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.317+0000 7fa44b7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4441a0090 con 0x7fa4441080f0 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50614/0 (socket says 172.21.15.23:50614) 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 -- 172.21.15.23:0/3413812854 learned_addr learned my addr 172.21.15.23:0/3413812854 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa449d9c700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 0x7fa44419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa448d9a700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444074b30 0x7fa444198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 -- 172.21.15.23:0/3413812854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 msgr2=0x7fa44419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 0x7fa44419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 -- 172.21.15.23:0/3413812854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444074b30 msgr2=0x7fa444198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444074b30 0x7fa444198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 -- 172.21.15.23:0/3413812854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400095f0 con 0x7fa444073550 2023-11-17T00:01:08.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.318+0000 7fa44959b700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa4400098e0 tx=0x7fa440004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.319+0000 7fa43a7fc700 1 -- 172.21.15.23:0/3413812854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44001d070 con 0x7fa444073550 2023-11-17T00:01:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.319+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44410de00 con 0x7fa444073550 2023-11-17T00:01:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.319+0000 7fa43a7fc700 1 -- 172.21.15.23:0/3413812854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4400054f0 con 0x7fa444073550 2023-11-17T00:01:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.319+0000 7fa43a7fc700 1 -- 172.21.15.23:0/3413812854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440004be0 con 0x7fa444073550 2023-11-17T00:01:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.319+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44410dc70 con 0x7fa444073550 2023-11-17T00:01:08.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.322+0000 7fa43a7fc700 1 -- 172.21.15.23:0/3413812854 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa44000fbc0 con 0x7fa444073550 2023-11-17T00:01:08.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.323+0000 7fa45f067b80 1 Processor -- start 2023-11-17T00:01:08.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.323+0000 7fa43a7fc700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 0x7fa4300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.323+0000 7fa45f067b80 1 -- start start 2023-11-17T00:01:08.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.323+0000 7fa43a7fc700 1 -- 172.21.15.23:0/3413812854 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4400cdea0 con 0x7fa444073550 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.323+0000 7fa449d9c700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 0x7fa4300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf49fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4a500 0x5564fdf4aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 0x5564fdf4b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fde4f010 con 0x5564fdf4b0c0 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdea1850 con 0x5564fdf49bb0 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa45f067b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fde9ace0 con 0x5564fdf4a500 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa449d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 0x5564fdf4b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf49fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa448d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4a500 0x5564fdf4aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa449d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 0x5564fdf4b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50618/0 (socket says 172.21.15.23:50618) 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa449d9c700 1 -- 172.21.15.23:0/3202523876 learned_addr learned my addr 172.21.15.23:0/3202523876 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 -- 172.21.15.23:0/3202523876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4a500 msgr2=0x5564fdf4aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4a500 0x5564fdf4aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 -- 172.21.15.23:0/3202523876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 msgr2=0x5564fdf4b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 0x5564fdf4b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.324+0000 7fa44959b700 1 -- 172.21.15.23:0/3202523876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564fdd1c5e0 con 0x5564fdf49bb0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.325+0000 7fa44959b700 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf49fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa440000c00 tx=0x7fa440005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bab758edcb8d86e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.325+0000 7fa449d9c700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 0x7fa4300a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa43c004290 tx=0x7fa43c003f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.325+0000 7fa42f7fe700 1 -- 172.21.15.23:0/3202523876 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44009f000 con 0x5564fdf49bb0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.325+0000 7fa42f7fe700 1 -- 172.21.15.23:0/3202523876 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4400a3390 con 0x5564fdf49bb0 2023-11-17T00:01:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.325+0000 7fa42f7fe700 1 -- 172.21.15.23:0/3202523876 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44009a720 con 0x5564fdf49bb0 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- 172.21.15.23:0/3202523876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 msgr2=0x5564fdf49fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf49fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa440000c00 tx=0x7fa440005ef0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- 172.21.15.23:0/3202523876 shutdown_connections 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4b0c0 0x5564fdf4b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf49fc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 --2- 172.21.15.23:0/3202523876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4a500 0x5564fdf4aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- 172.21.15.23:0/3202523876 >> 172.21.15.23:0/3202523876 conn(0x5564fdf3f310 msgr2=0x5564fdf44620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- 172.21.15.23:0/3202523876 shutdown_connections 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- 172.21.15.23:0/3202523876 wait complete. 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 -- start start 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.326+0000 7fa45f067b80 1 Processor -- start 2023-11-17T00:01:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf5a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fde3c890 con 0x5564fdf4a500 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fde3c530 con 0x5564fdf49bb0 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdd1e8d0 con 0x5564fdf4b0c0 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa45f067b80 10 client.0 _ll_register_callbacks cb 0x5564fde5a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa448d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa448d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50632/0 (socket says 172.21.15.23:50632) 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa449d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf5a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa448d9a700 1 -- 172.21.15.23:0/2303351203 learned_addr learned my addr 172.21.15.23:0/2303351203 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa44959b700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa448d9a700 1 -- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 msgr2=0x5564fdf5a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.327+0000 7fa448d9a700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf5a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa448d9a700 1 -- 172.21.15.23:0/2303351203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 msgr2=0x5564fdf531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa448d9a700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa448d9a700 1 -- 172.21.15.23:0/2303351203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44410dc70 con 0x5564fdf4a500 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa448d9a700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa434011780 tx=0x7fa434011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b8722d09c05cb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa42cff9700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434016070 con 0x5564fdf4a500 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564fdf3cad0 con 0x5564fdf4a500 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa434004320 con 0x5564fdf4a500 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4340044e0 con 0x5564fdf4a500 2023-11-17T00:01:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa45f067b80 10 client.15285 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.328+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5564fde61140 con 0x5564fdf4a500 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564fdf3cad0 con 0x5564fdf4a500 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa4340055f0 con 0x5564fdf4a500 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 10 client.15285 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 10 client.15285 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 20 client.15285 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdf5e490 0x5564fdf5d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 4 client.15285 mds_command: new command op to 24529 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 --> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564fde5d050 con 0x5564fdf5e490 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.329+0000 7fa44959b700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdf5e490 0x5564fdf5d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.330+0000 7fa44959b700 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdf5e490 0x5564fdf5d840 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b0050e2177cbd54 server_cookie=66a331c2ada49b3f in_seq=0 out_seq=0 2023-11-17T00:01:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.330+0000 7fa42cff9700 10 client.15285 ms_handle_connect on v2:172.21.15.173:6834/405519546 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.330+0000 7fa42cff9700 1 -- 172.21.15.23:0/2303351203 <== mds.2 v2:172.21.15.173:6834/405519546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5564fde5d050 con 0x5564fdf5e490 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.330+0000 7fa42cff9700 10 client.15285 handle_command_reply: tid=0 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.330+0000 7fa45f067b80 1 client.15285 shutdown 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 msgr2=0x5564fdf53700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa434011780 tx=0x7fa434011a90 comp rx=0 tx=0).stop 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 shutdown_connections 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf5a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdf5e490 0x5564fdf5d840 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 --2- 172.21.15.23:0/2303351203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf53700 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 >> 172.21.15.23:0/2303351203 conn(0x5564fdf3f310 msgr2=0x5564fdf43d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa42cff9700 0 client.15285 ms_handle_reset on v2:172.21.15.173:6834/405519546 2023-11-17T00:01:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 shutdown_connections 2023-11-17T00:01:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 1 -- 172.21.15.23:0/2303351203 wait complete. 2023-11-17T00:01:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.331+0000 7fa45f067b80 20 client.15285 trim_cache size 0 max 16384 2023-11-17T00:01:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.349+0000 7fa45f067b80 1 Processor -- start 2023-11-17T00:01:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 -- start start 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 0x5564fdf5a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf67200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdf543b0 con 0x5564fdf49bb0 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdda6ed0 con 0x5564fdf4a500 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa45f067b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fde36e10 con 0x5564fdf4b0c0 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa448d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 0x5564fdf5a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50642/0 (socket says 172.21.15.23:50642) 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 -- 172.21.15.23:0/1249610280 learned_addr learned my addr 172.21.15.23:0/1249610280 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa449d9c700 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf67200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 -- 172.21.15.23:0/1249610280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 msgr2=0x5564fdf67200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf67200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.350+0000 7fa44959b700 1 -- 172.21.15.23:0/1249610280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 msgr2=0x5564fdf5a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa44959b700 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 0x5564fdf5a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa44959b700 1 -- 172.21.15.23:0/1249610280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564fde60e90 con 0x5564fdf49bb0 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa448d9a700 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 0x5564fdf5a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa44959b700 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa44009b630 tx=0x7fa440005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5327875743e5da4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa42ffff700 1 -- 172.21.15.23:0/1249610280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44009b4d0 con 0x5564fdf49bb0 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa42ffff700 1 -- 172.21.15.23:0/1249610280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa44009f000 con 0x5564fdf49bb0 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.351+0000 7fa42ffff700 1 -- 172.21.15.23:0/1249610280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44009a720 con 0x5564fdf49bb0 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- 172.21.15.23:0/1249610280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 msgr2=0x5564fdf51fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa44009b630 tx=0x7fa440005a90 comp rx=0 tx=0).stop 2023-11-17T00:01:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- 172.21.15.23:0/1249610280 shutdown_connections 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf67200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf49bb0 0x5564fdf51fd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1249610280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf4a500 0x5564fdf5a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- 172.21.15.23:0/1249610280 >> 172.21.15.23:0/1249610280 conn(0x5564fdf3f310 msgr2=0x5564fdf445f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- 172.21.15.23:0/1249610280 shutdown_connections 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- 172.21.15.23:0/1249610280 wait complete. 2023-11-17T00:01:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 -- start start 2023-11-17T00:01:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.352+0000 7fa45f067b80 1 Processor -- start 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf54e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdf5a250 con 0x5564fdf4a500 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdf5a3c0 con 0x5564fdf49bb0 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fdf5a580 con 0x5564fdf4b0c0 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa45f067b80 10 client.0 _ll_register_callbacks cb 0x7fa434009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa448d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf54e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57504/0 (socket says 172.21.15.23:57504) 2023-11-17T00:01:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 -- 172.21.15.23:0/1511815074 learned_addr learned my addr 172.21.15.23:0/1511815074 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa449d9c700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 -- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 msgr2=0x5564fdf59d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf59d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 -- 172.21.15.23:0/1511815074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 msgr2=0x5564fdf54e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf54e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.353+0000 7fa44959b700 1 -- 172.21.15.23:0/1511815074 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564fde61140 con 0x5564fdf49bb0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa44959b700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa4400a2e60 tx=0x7fa440009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b193e5ce835bdc88 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44009b4d0 con 0x5564fdf49bb0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4400cb300 con 0x5564fdf49bb0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4400a3390 con 0x5564fdf49bb0 2023-11-17T00:01:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4400cb500 con 0x5564fdf49bb0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa45f067b80 10 client.25310 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5564fdd1c5e0 con 0x5564fdf49bb0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4400a3390 con 0x5564fdf49bb0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa440096dc0 con 0x5564fdf49bb0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.354+0000 7fa45f067b80 10 client.25310 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 10 client.25310 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 20 client.25310 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdfb64a0 0x5564fdfb2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 4 client.25310 mds_command: new command op to 24529 tid=0[{"prefix": "status"}] 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 --> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5564fde69610 con 0x5564fdfb64a0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa448d9a700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdfb64a0 0x5564fdfb2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa42cff9700 10 client.25310 ms_handle_connect on v2:172.21.15.173:6834/405519546 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa448d9a700 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdfb64a0 0x5564fdfb2e70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3543cf9f18d91ae5 server_cookie=67263938e05625e1 in_seq=0 out_seq=0 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa42cff9700 1 -- 172.21.15.23:0/1511815074 <== mds.2 v2:172.21.15.173:6834/405519546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5564fde69610 con 0x5564fdfb64a0 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa42cff9700 10 client.25310 handle_command_reply: tid=0 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.355+0000 7fa45f067b80 1 client.25310 shutdown 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 msgr2=0x5564fdf54920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa4400a2e60 tx=0x7fa440009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 shutdown_connections 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564fdf4b0c0 0x5564fdf59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5564fdf49bb0 0x5564fdf54920 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.173:6834/405519546,v1:172.21.15.173:6835/405519546] conn(0x5564fdfb64a0 0x5564fdfb2e70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 --2- 172.21.15.23:0/1511815074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5564fdf4a500 0x5564fdf54e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 >> 172.21.15.23:0/1511815074 conn(0x5564fdf3f310 msgr2=0x5564fdf5d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa42cff9700 0 client.25310 ms_handle_reset on v2:172.21.15.173:6834/405519546 2023-11-17T00:01:08.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 shutdown_connections 2023-11-17T00:01:08.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 1 -- 172.21.15.23:0/1511815074 wait complete. 2023-11-17T00:01:08.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa45f067b80 20 client.25310 trim_cache size 0 max 16384 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24529, 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 94.652167559000006, 2023-11-17T00:01:08.365 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:01:08.366 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:01:08.366 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:01:08.366 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 162.75233381800001 2023-11-17T00:01:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 msgr2=0x7fa4300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 0x7fa4300a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa43c004290 tx=0x7fa43c003f00 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 msgr2=0x7fa444198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.356+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa4400098e0 tx=0x7fa440004730 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 shutdown_connections 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fa4300a1150 0x7fa4300a3600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4441080f0 0x7fa44419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa444073550 0x7fa444198240 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 --2- 172.21.15.23:0/3413812854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa444074b30 0x7fa444198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 >> 172.21.15.23:0/3413812854 conn(0x7fa4440fb810 msgr2=0x7fa4440fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 shutdown_connections 2023-11-17T00:01:08.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.357+0000 7fa44b7ff700 1 -- 172.21.15.23:0/3413812854 wait complete. 2023-11-17T00:01:08.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:08 smithi023 ceph-mon[97824]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:08.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:08 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/429860594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.374 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:08 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/913429548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.377 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-17T00:01:08.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 -- 172.21.15.23:0/4029082237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 msgr2=0x7f02980739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 --2- 172.21.15.23:0/4029082237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02980739c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f028c009990 tx=0x7f028c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 -- 172.21.15.23:0/4029082237 shutdown_connections 2023-11-17T00:01:08.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 --2- 172.21.15.23:0/4029082237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029810a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 --2- 172.21.15.23:0/4029082237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 0x7f0298072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 --2- 172.21.15.23:0/4029082237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02980739c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 -- 172.21.15.23:0/4029082237 >> 172.21.15.23:0/4029082237 conn(0x7f02980fb810 msgr2=0x7f02980fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 -- 172.21.15.23:0/4029082237 shutdown_connections 2023-11-17T00:01:08.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.594+0000 7f029e5ff700 1 -- 172.21.15.23:0/4029082237 wait complete. 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.595+0000 7f029e5ff700 1 Processor -- start 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 -- start start 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 0x7f0298198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029819fd70 con 0x7f0298073550 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029819feb0 con 0x7f0298074b30 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029e5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02981a0040 con 0x7f02981080f0 2023-11-17T00:01:08.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f0297fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f029cb9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f0297fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50680/0 (socket says 172.21.15.23:50680) 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f0297fff700 1 -- 172.21.15.23:0/444434453 learned_addr learned my addr 172.21.15.23:0/444434453 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.596+0000 7f02977fe700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 0x7f0298198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 -- 172.21.15.23:0/444434453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 msgr2=0x7f0298198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 0x7f0298198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 -- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 msgr2=0x7f02981981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 -- 172.21.15.23:0/444434453 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0095f0 con 0x7f02981080f0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f0297fff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029cb9c700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029819f7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f029000efc0 tx=0x7f029000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f02957fa700 1 -- 172.21.15.23:0/444434453 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0290018070 con 0x7f02981080f0 2023-11-17T00:01:08.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f02957fa700 1 -- 172.21.15.23:0/444434453 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f029000f040 con 0x7f02981080f0 2023-11-17T00:01:08.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029810de10 con 0x7f02981080f0 2023-11-17T00:01:08.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f02957fa700 1 -- 172.21.15.23:0/444434453 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029000cd40 con 0x7f02981080f0 2023-11-17T00:01:08.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.597+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029810dc80 con 0x7f02981080f0 2023-11-17T00:01:08.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.601+0000 7f02957fa700 1 -- 172.21.15.23:0/444434453 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0290003a00 con 0x7f02981080f0 2023-11-17T00:01:08.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 Processor -- start 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 -- start start 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df7bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df7cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7de81010 con 0x557d7df7c500 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.602+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7ded3850 con 0x557d7df7bbb0 2023-11-17T00:01:08.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7deccce0 con 0x557d7df7d0c0 2023-11-17T00:01:08.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02957fa700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 0x7f02800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df7bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f0297fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df7cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52810/0 (socket says 172.21.15.23:52810) 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 -- 172.21.15.23:0/3979715391 learned_addr learned my addr 172.21.15.23:0/3979715391 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02957fa700 1 -- 172.21.15.23:0/444434453 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02900cc6f0 con 0x7f02981080f0 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 -- 172.21.15.23:0/3979715391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 msgr2=0x557d7df7bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df7bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 -- 172.21.15.23:0/3979715391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 msgr2=0x557d7df7cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df7cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.603+0000 7f02977fe700 1 -- 172.21.15.23:0/3979715391 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7dd4e5e0 con 0x557d7df7d0c0 2023-11-17T00:01:08.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.604+0000 7f02977fe700 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f028c005cd0 tx=0x7f028c005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccd0b434008e159c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.604+0000 7f029cb9c700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 0x7f02800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.604+0000 7f027e7fc700 1 -- 172.21.15.23:0/3979715391 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00bc20 con 0x557d7df7d0c0 2023-11-17T00:01:08.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.604+0000 7f027e7fc700 1 -- 172.21.15.23:0/3979715391 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f028c01e070 con 0x557d7df7d0c0 2023-11-17T00:01:08.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.604+0000 7f027e7fc700 1 -- 172.21.15.23:0/3979715391 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c017650 con 0x557d7df7d0c0 2023-11-17T00:01:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:08 smithi156 ceph-mon[131105]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:08.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:08 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/429860594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:08 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/913429548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:08 smithi173 ceph-mon[131534]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:08.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:08 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/429860594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:08 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/913429548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3979715391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 msgr2=0x557d7df7d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f029cb9c700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 0x7f02800a3680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0290009fd0 tx=0x7f0290000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f028c005cd0 tx=0x7f028c005fb0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3979715391 shutdown_connections 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df7d530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df7bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3979715391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df7cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3979715391 >> 172.21.15.23:0/3979715391 conn(0x557d7df71310 msgr2=0x557d7df76620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3979715391 shutdown_connections 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.605+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3979715391 wait complete. 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.606+0000 7f02b1eefb80 1 -- start start 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.606+0000 7f02b1eefb80 1 Processor -- start 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df85590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df85ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df8cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7de6e890 con 0x557d7df7c500 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7de6e530 con 0x557d7df7bbb0 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7dd508d0 con 0x557d7df7d0c0 2023-11-17T00:01:08.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02b1eefb80 10 client.0 _ll_register_callbacks cb 0x557d7de8c730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df8cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f0297fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df85ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df8cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52824/0 (socket says 172.21.15.23:52824) 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02977fe700 1 -- 172.21.15.23:0/3613384185 learned_addr learned my addr 172.21.15.23:0/3613384185 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.607+0000 7f02977fe700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df85590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 -- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 msgr2=0x557d7df8cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df8cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 -- 172.21.15.23:0/3613384185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 msgr2=0x557d7df85590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df85590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 -- 172.21.15.23:0/3613384185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c020040 con 0x557d7df7c500 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f02977fe700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df85590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f0297fff700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df85ad0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0288009690 tx=0x7f0288009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e352e1021f1ccbe4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f02777fe700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288017070 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f028800f970 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.608+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d7df6ead0 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f028800fb30 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 10 client.15303 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557d7de93140 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d7df6ead0 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0288003970 con 0x557d7df7c500 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 10 client.15303 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 20 client.15303 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:08.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7df90890 0x557d7df8fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 4 client.15303 mds_command: new command op to 24565 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.609+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 --> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d7de8f050 con 0x557d7df90890 2023-11-17T00:01:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.610+0000 7f02977fe700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7df90890 0x557d7df8fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.610+0000 7f02977fe700 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7df90890 0x557d7df8fcc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1eb3f5cff6cedf45 server_cookie=d4884fd830464894 in_seq=0 out_seq=0 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.610+0000 7f02777fe700 10 client.15303 ms_handle_connect on v2:172.21.15.173:6838/3306732798 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02777fe700 1 -- 172.21.15.23:0/3613384185 <== mds.1 v2:172.21.15.173:6838/3306732798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557d7de8f050 con 0x557d7df90890 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02777fe700 10 client.15303 handle_command_reply: tid=0 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 client.15303 shutdown 2023-11-17T00:01:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 msgr2=0x557d7df85ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df85ad0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0288009690 tx=0x7f0288009ef0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 shutdown_connections 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7df90890 0x557d7df8fcc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7d0c0 0x557d7df8cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02777fe700 0 client.15303 ms_handle_reset on v2:172.21.15.173:6838/3306732798 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7bbb0 0x557d7df85590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3613384185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df85ad0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.611+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 >> 172.21.15.23:0/3613384185 conn(0x557d7df71310 msgr2=0x557d7df75d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.612+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 shutdown_connections 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.612+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3613384185 wait complete. 2023-11-17T00:01:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.612+0000 7f02b1eefb80 20 client.15303 trim_cache size 0 max 16384 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 Processor -- start 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 -- start start 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df8cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df8fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7df77550 con 0x557d7df7c500 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7df78150 con 0x557d7df7d0c0 2023-11-17T00:01:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.635+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7de8f050 con 0x557d7df7bbb0 2023-11-17T00:01:08.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f0297fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df8cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52832/0 (socket says 172.21.15.23:52832) 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 -- 172.21.15.23:0/3790972918 learned_addr learned my addr 172.21.15.23:0/3790972918 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df8fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 -- 172.21.15.23:0/3790972918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 msgr2=0x557d7df8fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df8fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 -- 172.21.15.23:0/3790972918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 msgr2=0x557d7df8cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df8cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f02977fe700 1 -- 172.21.15.23:0/3790972918 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7de92e90 con 0x557d7df7bbb0 2023-11-17T00:01:08.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.636+0000 7f0297fff700 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df8cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:08.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.637+0000 7f02977fe700 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f028c00fe60 tx=0x7f028c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ff0f6ade9fe9e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.637+0000 7f027effd700 1 -- 172.21.15.23:0/3790972918 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c02d040 con 0x557d7df7bbb0 2023-11-17T00:01:08.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.637+0000 7f027effd700 1 -- 172.21.15.23:0/3790972918 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f028c042070 con 0x557d7df7bbb0 2023-11-17T00:01:08.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.637+0000 7f027effd700 1 -- 172.21.15.23:0/3790972918 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c03b260 con 0x557d7df7bbb0 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.637+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3790972918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 msgr2=0x557d7df84240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f028c00fe60 tx=0x7f028c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3790972918 shutdown_connections 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df8fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7bbb0 0x557d7df84240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/3790972918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7c500 0x557d7df8cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3790972918 >> 172.21.15.23:0/3790972918 conn(0x557d7df71310 msgr2=0x557d7df995e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3790972918 shutdown_connections 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 -- 172.21.15.23:0/3790972918 wait complete. 2023-11-17T00:01:08.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 -- start start 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.638+0000 7f02b1eefb80 1 Processor -- start 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7df87ef0 con 0x557d7df7bbb0 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7dfe4710 con 0x557d7df7d0c0 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7dfe4850 con 0x557d7df7c500 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02b1eefb80 10 client.0 _ll_register_callbacks cb 0x7f02880127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f0297fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50708/0 (socket says 172.21.15.23:50708) 2023-11-17T00:01:08.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02977fe700 1 -- 172.21.15.23:0/1822779879 learned_addr learned my addr 172.21.15.23:0/1822779879 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f0297fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52848/0 (socket says 172.21.15.23:52848) 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.639+0000 7f02977fe700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 -- 172.21.15.23:0/1822779879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 msgr2=0x557d7df878d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df878d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 -- 172.21.15.23:0/1822779879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 msgr2=0x557d7df87390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 -- 172.21.15.23:0/1822779879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d7de93140 con 0x557d7df7c500 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02977fe700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f0297fff700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0288003dd0 tx=0x7f0288009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c24ebd3e75c0294 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02777fe700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288017040 con 0x557d7df7c500 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f028800fd80 con 0x557d7df7c500 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f028c01e070 con 0x557d7df7c500 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02880050c0 con 0x557d7df7c500 2023-11-17T00:01:08.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02b1eefb80 10 client.25360 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.640+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557d7dd4e5e0 con 0x557d7df7c500 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f028c01e070 con 0x557d7df7c500 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0288010b90 con 0x557d7df7c500 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 10 client.25360 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 10 client.25360 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 20 client.25360 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7dfe94f0 0x557d7dfe5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 4 client.25360 mds_command: new command op to 24565 tid=0[{"prefix": "status"}] 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 --> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557d7de9b610 con 0x557d7dfe94f0 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.641+0000 7f02977fe700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7dfe94f0 0x557d7dfe5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02777fe700 10 client.25360 ms_handle_connect on v2:172.21.15.173:6838/3306732798 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02977fe700 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7dfe94f0 0x557d7dfe5e80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7f5b75049d8ad31 server_cookie=c5223a83ffd9d0db in_seq=0 out_seq=0 2023-11-17T00:01:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02777fe700 1 -- 172.21.15.23:0/1822779879 <== mds.1 v2:172.21.15.173:6838/3306732798 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x557d7de9b610 con 0x557d7dfe94f0 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02777fe700 10 client.25360 handle_command_reply: tid=0 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 client.25360 shutdown 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 msgr2=0x557d7df8c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0288003dd0 tx=0x7f0288009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 shutdown_connections 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:6838/3306732798,v1:172.21.15.173:6839/3306732798] conn(0x557d7dfe94f0 0x557d7dfe5e80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7df7d0c0 0x557d7df878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d7df7bbb0 0x557d7df87390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 --2- 172.21.15.23:0/1822779879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d7df7c500 0x557d7df8c8d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 >> 172.21.15.23:0/1822779879 conn(0x557d7df71310 msgr2=0x557d7df84240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.642+0000 7f02777fe700 0 client.25360 ms_handle_reset on v2:172.21.15.173:6838/3306732798 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 shutdown_connections 2023-11-17T00:01:08.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f02b1eefb80 1 -- 172.21.15.23:0/1822779879 wait complete. 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f02b1eefb80 20 client.25360 trim_cache size 0 max 16384 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24565, 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:01:08.666 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 95.920120788000006, 2023-11-17T00:01:08.667 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:01:08.667 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:01:08.667 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:01:08.667 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 156.31962647899999 2023-11-17T00:01:08.667 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:01:08.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 msgr2=0x7f02800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 0x7f02800a3680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0290009fd0 tx=0x7f0290000f40 comp rx=0 tx=0).stop 2023-11-17T00:01:08.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 msgr2=0x7f029819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.643+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029819f7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f029000efc0 tx=0x7f029000c3a0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 shutdown_connections 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f02800a11d0 0x7f02800a3680 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02981080f0 0x7f029819f7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0298073550 0x7f02981981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 --2- 172.21.15.23:0/444434453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0298074b30 0x7f0298198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 >> 172.21.15.23:0/444434453 conn(0x7f02980fb810 msgr2=0x7f02980fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 shutdown_connections 2023-11-17T00:01:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.644+0000 7f029e5ff700 1 -- 172.21.15.23:0/444434453 wait complete. 2023-11-17T00:01:08.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-17T00:01:08.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 -- 172.21.15.23:0/2573102443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101450 msgr2=0x7f39b01018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 --2- 172.21.15.23:0/2573102443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101450 0x7f39b01018a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009990 tx=0x7f39a4009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 -- 172.21.15.23:0/2573102443 shutdown_connections 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 --2- 172.21.15.23:0/2573102443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101de0 0x7f39b01086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 --2- 172.21.15.23:0/2573102443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b0100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 --2- 172.21.15.23:0/2573102443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101450 0x7f39b01018a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 -- 172.21.15.23:0/2573102443 >> 172.21.15.23:0/2573102443 conn(0x7f39b00fb880 msgr2=0x7f39b00fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 -- 172.21.15.23:0/2573102443 shutdown_connections 2023-11-17T00:01:08.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.889+0000 7f39b4bff700 1 -- 172.21.15.23:0/2573102443 wait complete. 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.890+0000 7f39b4bff700 1 Processor -- start 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.890+0000 7f39b4bff700 1 -- start start 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101450 0x7f39b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101de0 0x7f39b019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019fd20 con 0x7f39b0101de0 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019fe60 con 0x7f39b0101450 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39b4bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019fff0 con 0x7f39b0100260 2023-11-17T00:01:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39add9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101450 0x7f39b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39aed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101de0 0x7f39b019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52868/0 (socket says 172.21.15.23:52868) 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.891+0000 7f39ae59c700 1 -- 172.21.15.23:0/872400121 learned_addr learned my addr 172.21.15.23:0/872400121 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 -- 172.21.15.23:0/872400121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101450 msgr2=0x7f39b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101450 0x7f39b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 -- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101de0 msgr2=0x7f39b019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101de0 0x7f39b019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 -- 172.21.15.23:0/872400121 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40095f0 con 0x7f39b0100260 2023-11-17T00:01:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39ae59c700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e970 tx=0x7f39a000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f399f7fe700 1 -- 172.21.15.23:0/872400121 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0009790 con 0x7f39b0100260 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f399f7fe700 1 -- 172.21.15.23:0/872400121 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39a000c7e0 con 0x7f39b0100260 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b010bc20 con 0x7f39b0100260 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f399f7fe700 1 -- 172.21.15.23:0/872400121 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0003c70 con 0x7f39b0100260 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.892+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b010ba90 con 0x7f39b0100260 2023-11-17T00:01:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.895+0000 7f399f7fe700 1 -- 172.21.15.23:0/872400121 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f39a000cc50 con 0x7f39b0100260 2023-11-17T00:01:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.897+0000 7f39c8458b80 1 Processor -- start 2023-11-17T00:01:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 -- start start 2023-11-17T00:01:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f399f7fe700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 0x7f39980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 0x5639e3d03fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d04ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 0x5639e3d05530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f399f7fe700 1 -- 172.21.15.23:0/872400121 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f39a0014070 con 0x7f39b0100260 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3c09010 con 0x5639e3d04500 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3c5b850 con 0x5639e3d050c0 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39c8458b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3c54ce0 con 0x5639e3d03bb0 2023-11-17T00:01:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39aed9d700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 0x7f39980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 0x5639e3d03fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.898+0000 7f39aed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 0x5639e3d05530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39aed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 0x5639e3d05530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57566/0 (socket says 172.21.15.23:57566) 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39aed9d700 1 -- 172.21.15.23:0/2963233974 learned_addr learned my addr 172.21.15.23:0/2963233974 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39add9b700 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d04ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 -- 172.21.15.23:0/2963233974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 msgr2=0x5639e3d05530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 0x5639e3d05530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 -- 172.21.15.23:0/2963233974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 msgr2=0x5639e3d04ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d04ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 -- 172.21.15.23:0/2963233974 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639e3ad65e0 con 0x5639e3d03bb0 2023-11-17T00:01:08.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 0x5639e3d03fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f39a0099bd0 tx=0x7f39a0003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13488f5db456e09b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f39aed9d700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 0x7f39980a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39a8007b90 tx=0x7f39a8014470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f399cff9700 1 -- 172.21.15.23:0/2963233974 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a009df00 con 0x5639e3d03bb0 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f399cff9700 1 -- 172.21.15.23:0/2963233974 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39a00a2320 con 0x5639e3d03bb0 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.899+0000 7f399cff9700 1 -- 172.21.15.23:0/2963233974 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00993c0 con 0x5639e3d03bb0 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 -- 172.21.15.23:0/2963233974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 msgr2=0x5639e3d03fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 0x5639e3d03fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f39a0099bd0 tx=0x7f39a0003710 comp rx=0 tx=0).stop 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 -- 172.21.15.23:0/2963233974 shutdown_connections 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d050c0 0x5639e3d05530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d03bb0 0x5639e3d03fc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2963233974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d04ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 -- 172.21.15.23:0/2963233974 >> 172.21.15.23:0/2963233974 conn(0x5639e3cf9310 msgr2=0x5639e3cfe620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 -- 172.21.15.23:0/2963233974 shutdown_connections 2023-11-17T00:01:08.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.900+0000 7f39c8458b80 1 -- 172.21.15.23:0/2963233974 wait complete. 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.901+0000 7f39c8458b80 1 -- start start 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.901+0000 7f39c8458b80 1 Processor -- start 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d04500 0x5639e3d0dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 0x5639e3d14af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3bf6890 con 0x5639e3d050c0 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3bf6530 con 0x5639e3d03bb0 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3ad88d0 con 0x5639e3d04500 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39c8458b80 10 client.0 _ll_register_callbacks cb 0x5639e3c14730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57572/0 (socket says 172.21.15.23:57572) 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 -- 172.21.15.23:0/2951289913 learned_addr learned my addr 172.21.15.23:0/2951289913 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39add9b700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d04500 0x5639e3d0dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39aed9d700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 0x5639e3d14af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 -- 172.21.15.23:0/2951289913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d04500 msgr2=0x5639e3d0dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d04500 0x5639e3d0dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 -- 172.21.15.23:0/2951289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 msgr2=0x5639e3d14af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 0x5639e3d14af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39ae59c700 1 -- 172.21.15.23:0/2951289913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b010ba90 con 0x5639e3d03bb0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.902+0000 7f39aed9d700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 0x5639e3d14af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4b0 tx=0x7f39a00966b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa853e69798353e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00993c0 con 0x5639e3d03bb0 2023-11-17T00:01:08.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f39a0099520 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639e3cf6ad0 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39a00adbd0 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f39c8458b80 10 client.25337 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5639e3c1b140 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39a0063080 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39a0094ae0 con 0x5639e3d03bb0 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.903+0000 7f39c8458b80 10 client.25337 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39c8458b80 10 client.25337 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39c8458b80 20 client.25337 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d18430 0x5639e3d17a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39c8458b80 4 client.25337 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:08.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639e3c17050 con 0x5639e3d18430 2023-11-17T00:01:08.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39add9b700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d18430 0x5639e3d17a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f3995ffb700 10 client.25337 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:08.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.904+0000 7f39add9b700 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d18430 0x5639e3d17a20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc2cb9eb483a447 server_cookie=4e32b89f0f2411c5 in_seq=0 out_seq=0 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f3995ffb700 1 -- 172.21.15.23:0/2951289913 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5639e3c17050 con 0x5639e3d18430 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f3995ffb700 10 client.25337 handle_command_reply: tid=0 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 client.25337 shutdown 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 msgr2=0x5639e3d0d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4b0 tx=0x7f39a00966b0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 shutdown_connections 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d18430 0x5639e3d17a20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d050c0 0x5639e3d14af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d0d570 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f3995ffb700 0 client.25337 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2951289913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d04500 0x5639e3d0dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 >> 172.21.15.23:0/2951289913 conn(0x5639e3cf9310 msgr2=0x5639e3cfdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 shutdown_connections 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.905+0000 7f39c8458b80 1 -- 172.21.15.23:0/2951289913 wait complete. 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.906+0000 7f39c8458b80 20 client.25337 trim_cache size 0 max 16384 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 Processor -- start 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 -- start start 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d14a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d08470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3cff6c0 con 0x5639e3d04500 2023-11-17T00:01:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3c17050 con 0x5639e3d03bb0 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.925+0000 7f39c8458b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3cf9120 con 0x5639e3d050c0 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57586/0 (socket says 172.21.15.23:57586) 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 -- 172.21.15.23:0/2194815451 learned_addr learned my addr 172.21.15.23:0/2194815451 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39add9b700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d14a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39aed9d700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d08470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 -- 172.21.15.23:0/2194815451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 msgr2=0x5639e3d08470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d08470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 -- 172.21.15.23:0/2194815451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 msgr2=0x5639e3d14a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d14a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:08.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 -- 172.21.15.23:0/2194815451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639e3c1ae90 con 0x5639e3d03bb0 2023-11-17T00:01:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39add9b700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d14a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.926+0000 7f39ae59c700 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4e0 tx=0x7f39a0098480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3d13baa8c38879 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f399d7fa700 1 -- 172.21.15.23:0/2194815451 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00a2130 con 0x5639e3d03bb0 2023-11-17T00:01:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f399d7fa700 1 -- 172.21.15.23:0/2194815451 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5639e3cf6ad0 con 0x5639e3d03bb0 2023-11-17T00:01:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f399d7fa700 1 -- 172.21.15.23:0/2194815451 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00adbd0 con 0x5639e3d03bb0 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 -- 172.21.15.23:0/2194815451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 msgr2=0x5639e3d07f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4e0 tx=0x7f39a0098480 comp rx=0 tx=0).stop 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 -- 172.21.15.23:0/2194815451 shutdown_connections 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d08470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d03bb0 0x5639e3d07f30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 --2- 172.21.15.23:0/2194815451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d04500 0x5639e3d14a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.927+0000 7f39c8458b80 1 -- 172.21.15.23:0/2194815451 >> 172.21.15.23:0/2194815451 conn(0x5639e3cf9310 msgr2=0x5639e3d0b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.928+0000 7f39c8458b80 1 -- 172.21.15.23:0/2194815451 shutdown_connections 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.928+0000 7f39c8458b80 1 -- 172.21.15.23:0/2194815451 wait complete. 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.928+0000 7f39c8458b80 1 -- start start 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.928+0000 7f39c8458b80 1 Processor -- start 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.928+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 0x5639e3d0ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d6c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3d105f0 con 0x5639e3d03bb0 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3d6c7f0 con 0x5639e3d04500 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639e3d6c960 con 0x5639e3d050c0 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39c8458b80 10 client.0 _ll_register_callbacks cb 0x7f39a80149e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39add9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 0x5639e3d0ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39aed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d6c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50766/0 (socket says 172.21.15.23:50766) 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 -- 172.21.15.23:0/3077381980 learned_addr learned my addr 172.21.15.23:0/3077381980 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39add9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 0x5639e3d0ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:57588/0 (socket says 172.21.15.23:57588) 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 -- 172.21.15.23:0/3077381980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 msgr2=0x5639e3d6c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d6c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 -- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 msgr2=0x5639e3d0ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 0x5639e3d0ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.082 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 -- 172.21.15.23:0/3077381980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0108fd0 con 0x5639e3d03bb0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f39ae59c700 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4b0 tx=0x7f39a0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9048b034e7306ddd server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.929+0000 7f3995ffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00ca490 con 0x5639e3d03bb0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f39a00b2ab0 con 0x5639e3d03bb0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39a00a2320 con 0x5639e3d03bb0 2023-11-17T00:01:09.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39a00adbd0 con 0x5639e3d03bb0 2023-11-17T00:01:09.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 10 client.15327 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:09.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5639e3c1b140 con 0x5639e3d03bb0 2023-11-17T00:01:09.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39a00a2320 con 0x5639e3d03bb0 2023-11-17T00:01:09.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f39a007ad90 con 0x5639e3d03bb0 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 10 client.15327 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 20 client.15327 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.930+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d71010 0x5639e3d6db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f39c8458b80 4 client.15327 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2023-11-17T00:01:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5639e3c23610 con 0x5639e3d71010 2023-11-17T00:01:09.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f39add9b700 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d71010 0x5639e3d6db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f3995ffb700 10 client.15327 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f39add9b700 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d71010 0x5639e3d6db90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a43a2d15403e12 server_cookie=3337ed0d6c79263d in_seq=0 out_seq=0 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f3995ffb700 1 -- 172.21.15.23:0/3077381980 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5639e3c23610 con 0x5639e3d71010 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f3995ffb700 10 client.15327 handle_command_reply: tid=0 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.931+0000 7f39c8458b80 1 client.15327 shutdown 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 msgr2=0x5639e3d0fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c4b0 tx=0x7f39a0003680 comp rx=0 tx=0).stop 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 shutdown_connections 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5639e3d050c0 0x5639e3d6c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5639e3d71010 0x5639e3d6db90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639e3d03bb0 0x5639e3d0fa10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 --2- 172.21.15.23:0/3077381980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5639e3d04500 0x5639e3d0ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f3995ffb700 0 client.15327 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 >> 172.21.15.23:0/3077381980 conn(0x5639e3cf9310 msgr2=0x5639e3d0d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 shutdown_connections 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 1 -- 172.21.15.23:0/3077381980 wait complete. 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.932+0000 7f39c8458b80 20 client.15327 trim_cache size 0 max 16384 2023-11-17T00:01:09.241 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:01:09.242 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "d04064ea-84da-11ee-8dbe-212e2dc638e7", 2023-11-17T00:01:09.242 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2023-11-17T00:01:09.242 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24572, 2023-11-17T00:01:09.242 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-11-17T00:01:09.244 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-11-17T00:01:09.244 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-11-17T00:01:09.283 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 142.67264795700001, 2023-11-17T00:01:09.283 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 24, 2023-11-17T00:01:09.283 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2023-11-17T00:01:09.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 88, 2023-11-17T00:01:09.284 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 147.87143620800001 2023-11-17T00:01:09.284 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:01:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 msgr2=0x7f39980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 0x7f39980a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39a8007b90 tx=0x7f39a8014470 comp rx=0 tx=0).stop 2023-11-17T00:01:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 msgr2=0x7f39b01981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e970 tx=0x7f39a000ec80 comp rx=0 tx=0).stop 2023-11-17T00:01:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 shutdown_connections 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b0101de0 0x7f39b019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f39980a1150 0x7f39980a3600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39b0100260 0x7f39b01981a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 --2- 172.21.15.23:0/872400121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39b0101450 0x7f39b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 >> 172.21.15.23:0/872400121 conn(0x7f39b00fb880 msgr2=0x7f39b0106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 shutdown_connections 2023-11-17T00:01:09.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:08.933+0000 7f39b4bff700 1 -- 172.21.15.23:0/872400121 wait complete. 2023-11-17T00:01:09.287 INFO:teuthology.run_tasks:Running task kclient... 2023-11-17T00:01:09.336 INFO:tasks.kclient:Mounting kernel clients... 2023-11-17T00:01:09.337 INFO:tasks.kclient:config is {'client.0': {}} 2023-11-17T00:01:09.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:09.337 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-11-17T00:01:09.381 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-17T00:01:09.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:09.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-17T00:01:09.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:09.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2023-11-17T00:01:09.606 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2023-11-17T00:01:09.606 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-17T00:01:09.606 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:09.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-17T00:01:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:09 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:09 smithi023 ceph-mon[97824]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:09.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:09 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:09 smithi173 ceph-mon[131534]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:09 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:09.858 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:09 smithi156 ceph-mon[131105]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:01:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:10 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:10 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:10 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2311425233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 msgr2=0x7fd3e810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2311425233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e810d160 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000b0d0 tx=0x7fd3e000b3e0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2311425233 shutdown_connections 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2311425233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e810d160 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2311425233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2311425233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2311425233 >> 172.21.15.23:0/2311425233 conn(0x7fd3e80fbdc0 msgr2=0x7fd3e80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2311425233 shutdown_connections 2023-11-17T00:01:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.871+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2311425233 wait complete. 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.872+0000 7fd3efe6f700 1 Processor -- start 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.872+0000 7fd3efe6f700 1 -- start start 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e819c5d0 con 0x7fd3e81055e0 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e819c740 con 0x7fd3e8100ee0 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3efe6f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e819c900 con 0x7fd3e8100540 2023-11-17T00:01:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3edc0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3edc0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37096/0 (socket says 172.21.15.23:37096) 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3edc0b700 1 -- 172.21.15.23:0/2957219884 learned_addr learned my addr 172.21.15.23:0/2957219884 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3ed40a700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3ee40c700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.873+0000 7fd3edc0b700 1 -- 172.21.15.23:0/2957219884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 msgr2=0x7fd3e8198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3edc0b700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3edc0b700 1 -- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 msgr2=0x7fd3e819bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3edc0b700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e819bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3edc0b700 1 -- 172.21.15.23:0/2957219884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8009520 con 0x7fd3e8100540 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3ed40a700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3ee40c700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e819bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3edc0b700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400e9a0 tx=0x7fd3e400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4009790 con 0x7fd3e8100540 2023-11-17T00:01:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e0009d00 con 0x7fd3e8100540 2023-11-17T00:01:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3e4010430 con 0x7fd3e8100540 2023-11-17T00:01:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4018680 con 0x7fd3e8100540 2023-11-17T00:01:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.874+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e8110370 con 0x7fd3e8100540 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.875+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e81101e0 con 0x7fd3e8100540 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.877+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd3e4010810 con 0x7fd3e8100540 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.878+0000 7fd3deffd700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 0x7fd3d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.879+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3e4014070 con 0x7fd3e8100540 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.879+0000 7fd3ed40a700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 0x7fd3d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:11.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.879+0000 7fd3ed40a700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 0x7fd3d40a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8199cf0 tx=0x7fd3d8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:11.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:11.883+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd3e4051730 con 0x7fd3e8100540 2023-11-17T00:01:12.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:01:12.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:11 smithi023 ceph-mon[97824]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:12.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:01:12.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:11 smithi173 ceph-mon[131534]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:01:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:01:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:01:12.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:01:12.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:01:12.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:01:12.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:11 smithi156 ceph-mon[131105]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:12.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.137+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd3e8066ea0 con 0x7fd3e8100540 2023-11-17T00:01:12.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.137+0000 7fd3deffd700 1 -- 172.21.15.23:0/2957219884 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd3e4096270 con 0x7fd3e8100540 2023-11-17T00:01:12.139 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-17T00:01:12.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 msgr2=0x7fd3d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 0x7fd3d40a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8199cf0 tx=0x7fd3d8009260 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 msgr2=0x7fd3e81986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400e9a0 tx=0x7fd3e400ecb0 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 shutdown_connections 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3e81055e0 0x7fd3e819bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd3d40a1150 0x7fd3d40a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3e8100540 0x7fd3e81986d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 --2- 172.21.15.23:0/2957219884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3e8100ee0 0x7fd3e8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 >> 172.21.15.23:0/2957219884 conn(0x7fd3e80fbdc0 msgr2=0x7fd3e810b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:12.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 shutdown_connections 2023-11-17T00:01:12.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:01:12.140+0000 7fd3efe6f700 1 -- 172.21.15.23:0/2957219884 wait complete. 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-17T00:01:12.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-17T00:01:12.678 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-17T00:01:12.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:12.678 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2023-11-17T00:01:12.726 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-17T00:01:12.727 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41517sec preferred_lft 41517sec 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2023-11-17T00:01:12.728 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2023-11-17T00:01:12.729 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-17T00:01:12.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2023-11-17T00:01:12.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-17T00:01:12.730 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-17T00:01:12.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:12 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2957219884' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-17T00:01:12.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:12.979 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-17T00:01:13.049 INFO:teuthology.orchestra.run.smithi023.stdout:1 2023-11-17T00:01:13.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.053 DEBUG:teuthology.orchestra.run.smithi023:> route 2023-11-17T00:01:13.100 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2023-11-17T00:01:13.100 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-17T00:01:13.101 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-17T00:01:13.101 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-17T00:01:13.101 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-17T00:01:13.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-17T00:01:13.102 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-17T00:01:13.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:12 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2957219884' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-17T00:01:13.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:12 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2957219884' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-17T00:01:13.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.280 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-11-17T00:01:13.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.325 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2023-11-17T00:01:13.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.409 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-17T00:01:13.409 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-17T00:01:13.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-17T00:01:13.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-17T00:01:13.409 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-17T00:01:13.598 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-17T00:01:13.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-17T00:01:13.598 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-17T00:01:13.810 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:13 smithi023 ceph-mon[97824]: pgmap v371: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:13.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:13.853 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-11-17T00:01:13.854 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-11-17T00:01:13.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2023-11-17T00:01:13.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2023-11-17T00:01:13.854 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-11-17T00:01:14.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-17T00:01:14.029 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:14.073 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-17T00:01:14.074 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-17T00:01:14.074 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:14.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:13 smithi173 ceph-mon[131534]: pgmap v371: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:14.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:13 smithi156 ceph-mon[131105]: pgmap v371: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:01:14.158 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-11-17T00:01:14.158 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-11-17T00:01:14.159 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/406dd57e-732c-4c7f-a2d6-e1aab9b3632b 2023-11-17T00:01:14.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:14.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/406dd57e-732c-4c7f-a2d6-e1aab9b3632b /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-11-17T00:01:14.306 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-11-17T00:01:14.306 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2023-11-17T00:01:14.306 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2023-11-17T00:01:14.782 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-11-17T00:01:14.782 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2023-11-17T00:01:14.782 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2023-11-17T00:01:14.782 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.156:3300,172.21.15.173:3300" 2023-11-17T00:01:14.783 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.23:3300,172.21.15.156:3300,172.21.15.173:3300:/volumes/_nogroup/sv_1/406dd57e-732c-4c7f-a2d6-e1aab9b3632b 2023-11-17T00:01:14.783 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=d04064ea-84da-11ee-8dbe-212e2dc638e7" will pass to kernel 2023-11-17T00:01:14.790 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-17T00:01:14.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:14.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:14.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:14.864 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-17T00:01:14.952 INFO:teuthology.orchestra.run.smithi023.stdout:client25387 2023-11-17T00:01:14.953 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-17T00:01:14.953 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-17T00:01:14.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:14.998 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-17T00:01:15.125 INFO:teuthology.orchestra.run.smithi023.stdout:client25387 2023-11-17T00:01:15.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:15.125 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-17T00:01:15.214 INFO:teuthology.orchestra.run.smithi023.stdout:client25387 2023-11-17T00:01:15.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:01:15.215 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7.client25387/status') 2023-11-17T00:01:15.325 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.25387 (3)172.21.15.23:0/3777511179 2023-11-17T00:01:15.325 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2023-11-17T00:01:15.325 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2023-11-17T00:01:15.326 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2023-11-17T00:01:15.326 INFO:teuthology.orchestra.run.smithi023.stderr:71 bytes copied, 6.7285e-05 s, 1.1 MB/s 2023-11-17T00:01:15.329 INFO:teuthology.run_tasks:Running task check-counter... 2023-11-17T00:01:15.341 INFO:teuthology.run_tasks:Running task background_exec... 2023-11-17T00:01:15.353 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:01:15.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-11-17T00:01:15.359 INFO:teuthology.run_tasks:Running task workunit... 2023-11-17T00:01:15.374 INFO:tasks.workunit:Pulling workunits from ref 1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-17T00:01:15.374 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-17T00:01:15.374 INFO:tasks.workunit:timeout=3h 2023-11-17T00:01:15.374 INFO:tasks.workunit:cleanup=True 2023-11-17T00:01:15.375 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:cephfs_t:s0 2023-11-17T00:01:15.420 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-11-16 23:59:36.226299244 +0000 2023-11-17T00:01:15.421 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-11-16 23:59:36.226299244 +0000 2023-11-17T00:01:15.421 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-11-17 00:01:14.857701438 +0000 2023-11-17T00:01:15.421 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-11-16 23:59:36.226299244 +0000 2023-11-17T00:01:15.421 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-17T00:01:15.421 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-17T00:01:15.539 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/chrisphoffman/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c9565146c8647d4e00eb6c7ae626dc192680e7b 2023-11-17T00:01:15.587 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-17T00:01:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:16 smithi023 ceph-mon[97824]: pgmap v372: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:16 smithi173 ceph-mon[131534]: pgmap v372: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:16 smithi156 ceph-mon[131105]: pgmap v372: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-11-17T00:01:18.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:18 smithi023 ceph-mon[97824]: pgmap v373: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:18 smithi173 ceph-mon[131534]: pgmap v373: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:18 smithi156 ceph-mon[131105]: pgmap v373: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:20.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:19 smithi023 ceph-mon[97824]: pgmap v374: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:20.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:19 smithi156 ceph-mon[131105]: pgmap v374: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:20.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:19 smithi173 ceph-mon[131534]: pgmap v374: 129 pgs: 129 active+clean; 640 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-11-17T00:01:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:22 smithi023 ceph-mon[97824]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:01:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:22 smithi173 ceph-mon[131534]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:01:22.677 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:22 smithi156 ceph-mon[131105]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:01:24.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:24 smithi023 ceph-mon[97824]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 6 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:24 smithi156 ceph-mon[131105]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 6 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:26 smithi156 ceph-mon[131105]: pgmap v377: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:24 smithi173 ceph-mon[131534]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 6 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:26 smithi173 ceph-mon[131534]: pgmap v377: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:26 smithi023 ceph-mon[97824]: pgmap v377: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:28.586 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:28 smithi023 ceph-mon[97824]: pgmap v378: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:28 smithi173 ceph-mon[131534]: pgmap v378: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:28 smithi156 ceph-mon[131105]: pgmap v378: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:28.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2221898186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38103490 msgr2=0x7f3b38105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:28.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2221898186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38103490 0x7f3b38105870 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009990 tx=0x7f3b2c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2221898186 shutdown_connections 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2221898186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38105db0 0x7f3b38108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2221898186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38100b70 0x7f3b38102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2221898186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38103490 0x7f3b38105870 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.608+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2221898186 >> 172.21.15.23:0/2221898186 conn(0x7f3b380fa530 msgr2=0x7f3b380fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.609+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2221898186 shutdown_connections 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.609+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2221898186 wait complete. 2023-11-17T00:01:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 Processor -- start 2023-11-17T00:01:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 -- start start 2023-11-17T00:01:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 0x7f3b381963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 0x7f3b3819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3819d9e0 con 0x7f3b38100b70 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3819db20 con 0x7f3b38105db0 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b3c99d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3819dcb0 con 0x7f3b38103490 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b36ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 0x7f3b381963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42164/0 (socket says 172.21.15.23:42164) 2023-11-17T00:01:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.611+0000 7f3b377fe700 1 -- 172.21.15.23:0/2589605919 learned_addr learned my addr 172.21.15.23:0/2589605919 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 -- 172.21.15.23:0/2589605919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 msgr2=0x7f3b3819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b37fff700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 0x7f3b3819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 0x7f3b3819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 -- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 msgr2=0x7f3b38195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 -- 172.21.15.23:0/2589605919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0095f0 con 0x7f3b38103490 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b377fe700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b37fff700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 0x7f3b3819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 0x7f3b381963a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0098e0 tx=0x7f3b2c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b34ff9700 1 -- 172.21.15.23:0/2589605919 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c01d070 con 0x7f3b38103490 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b34ff9700 1 -- 172.21.15.23:0/2589605919 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b2c021410 con 0x7f3b38103490 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3810ba30 con 0x7f3b38103490 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b34ff9700 1 -- 172.21.15.23:0/2589605919 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c017650 con 0x7f3b38103490 2023-11-17T00:01:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.612+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3810b8a0 con 0x7f3b38103490 2023-11-17T00:01:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.617+0000 7f3b34ff9700 1 -- 172.21.15.23:0/2589605919 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3b2c021580 con 0x7f3b38103490 2023-11-17T00:01:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.618+0000 7f3b5149db80 1 Processor -- start 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 -- start start 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b34ff9700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 0x7f3b200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b34ff9700 1 -- 172.21.15.23:0/2589605919 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3b2c0cdff0 con 0x7f3b38103490 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c22287100 0x562c22287570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22188000 con 0x562c22288440 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c221da840 con 0x562c22287100 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b37fff700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 0x7f3b200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b5149db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c221d3cd0 con 0x562c220de6b0 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.619+0000 7f3b36ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:50414/0 (socket says 172.21.15.23:50414) 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b36ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42166/0 (socket says 172.21.15.23:42166) 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 -- 172.21.15.23:0/745298542 learned_addr learned my addr 172.21.15.23:0/745298542 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b37fff700 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c22287100 0x562c22287570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 -- 172.21.15.23:0/745298542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c22287100 msgr2=0x562c22287570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c22287100 0x562c22287570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 -- 172.21.15.23:0/745298542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 msgr2=0x562c22286bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 -- 172.21.15.23:0/745298542 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c220555d0 con 0x562c220de6b0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b37fff700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 0x7f3b200a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009460 tx=0x7f3b30009920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b377fe700 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011680 tx=0x7f3b28011990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de03339650566139 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.620+0000 7f3b1dffb700 1 -- 172.21.15.23:0/745298542 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28004030 con 0x562c220de6b0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b1dffb700 1 -- 172.21.15.23:0/745298542 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b28003680 con 0x562c220de6b0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b1dffb700 1 -- 172.21.15.23:0/745298542 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28019830 con 0x562c220de6b0 2023-11-17T00:01:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 -- 172.21.15.23:0/745298542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 msgr2=0x562c22287f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011680 tx=0x7f3b28011990 comp rx=0 tx=0).stop 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 -- 172.21.15.23:0/745298542 shutdown_connections 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c22287100 0x562c22287570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c220de6b0 0x562c22287f00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 --2- 172.21.15.23:0/745298542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22286bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.621+0000 7f3b5149db80 1 -- 172.21.15.23:0/745298542 >> 172.21.15.23:0/745298542 conn(0x562c222781a0 msgr2=0x562c2227d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.622+0000 7f3b5149db80 1 -- 172.21.15.23:0/745298542 shutdown_connections 2023-11-17T00:01:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.622+0000 7f3b5149db80 1 -- 172.21.15.23:0/745298542 wait complete. 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.622+0000 7f3b5149db80 1 -- start start 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.622+0000 7f3b5149db80 1 Processor -- start 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22290820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22297860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22175880 con 0x562c22288440 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22175520 con 0x562c220de6b0 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c220578c0 con 0x562c22287100 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b5149db80 10 client.0 _ll_register_callbacks cb 0x562c22193720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:45464/0 (socket says 172.21.15.23:45464) 2023-11-17T00:01:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22297860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b377fe700 1 -- 172.21.15.23:0/2440678126 learned_addr learned my addr 172.21.15.23:0/2440678126 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.623+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22290820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 -- 172.21.15.23:0/2440678126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 msgr2=0x562c22290820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22290820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 -- 172.21.15.23:0/2440678126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 msgr2=0x562c22297860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22297860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 -- 172.21.15.23:0/2440678126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b28009e10 con 0x562c220de6b0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b377fe700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b28014040 tx=0x7f3b28004210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ce95847c3aabc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.624+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28003e10 con 0x562c220de6b0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c22275960 con 0x562c220de6b0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3b28005040 con 0x562c220de6b0 2023-11-17T00:01:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b28005200 con 0x562c220de6b0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 10 client.25358 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562c2219a130 con 0x562c220de6b0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562c22275960 con 0x562c220de6b0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3b280195c0 con 0x562c220de6b0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 10 client.25358 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 10 client.25358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 20 client.25358 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c2229c010 0x562c2229e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.625+0000 7f3b5149db80 4 client.25358 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.626+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c22196040 con 0x562c2229c010 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.626+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c2229c010 0x562c2229e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.626+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c2229c010 0x562c2229e420 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5b192f0e3ab32d server_cookie=d2302719e5f1039b in_seq=0 out_seq=0 2023-11-17T00:01:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.626+0000 7f3b16ffd700 10 client.25358 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b16ffd700 1 -- 172.21.15.23:0/2440678126 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562c22196040 con 0x562c2229c010 2023-11-17T00:01:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b16ffd700 10 client.25358 handle_command_reply: tid=0 2023-11-17T00:01:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 client.25358 shutdown 2023-11-17T00:01:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 msgr2=0x562c222902e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b28014040 tx=0x7f3b28004210 comp rx=0 tx=0).stop 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 shutdown_connections 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c2229c010 0x562c2229e420 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c22297860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222902e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b16ffd700 0 client.25358 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 --2- 172.21.15.23:0/2440678126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22290820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.627+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 >> 172.21.15.23:0/2440678126 conn(0x562c222781a0 msgr2=0x562c2227ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.628+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 shutdown_connections 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.628+0000 7f3b5149db80 1 -- 172.21.15.23:0/2440678126 wait complete. 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.628+0000 7f3b5149db80 20 client.25358 trim_cache size 0 max 16384 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 Processor -- start 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 -- start start 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22297820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c2228f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2228f7d0 con 0x562c22288440 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2227de40 con 0x562c220de6b0 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b5149db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2228f5a0 con 0x562c22287100 2023-11-17T00:01:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c2228f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:45480/0 (socket says 172.21.15.23:45480) 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.646+0000 7f3b377fe700 1 -- 172.21.15.23:0/3181202077 learned_addr learned my addr 172.21.15.23:0/3181202077 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22297820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 -- 172.21.15.23:0/3181202077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 msgr2=0x562c22297820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22297820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 -- 172.21.15.23:0/3181202077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 msgr2=0x562c2228f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c2228f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 -- 172.21.15.23:0/3181202077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c22199e80 con 0x562c220de6b0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b377fe700 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3b280115a0 tx=0x7f3b28011990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe98d69e4e04fc9b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b1e7fc700 1 -- 172.21.15.23:0/3181202077 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2802ad80 con 0x562c220de6b0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b1e7fc700 1 -- 172.21.15.23:0/3181202077 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b280054f0 con 0x562c220de6b0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.647+0000 7f3b1e7fc700 1 -- 172.21.15.23:0/3181202077 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2802a600 con 0x562c220de6b0 2023-11-17T00:01:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 -- 172.21.15.23:0/3181202077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 msgr2=0x562c2228f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3b280115a0 tx=0x7f3b28011990 comp rx=0 tx=0).stop 2023-11-17T00:01:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 -- 172.21.15.23:0/3181202077 shutdown_connections 2023-11-17T00:01:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22288440 0x562c2228f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c2228f990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 --2- 172.21.15.23:0/3181202077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22287100 0x562c22297820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 -- 172.21.15.23:0/3181202077 >> 172.21.15.23:0/3181202077 conn(0x562c222781a0 msgr2=0x562c222a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 -- 172.21.15.23:0/3181202077 shutdown_connections 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.648+0000 7f3b5149db80 1 -- 172.21.15.23:0/3181202077 wait complete. 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 -- start start 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 Processor -- start 2023-11-17T00:01:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222959c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22288440 0x562c222964c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22296bd0 con 0x562c22287100 2023-11-17T00:01:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22296d40 con 0x562c220de6b0 2023-11-17T00:01:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c22296f00 con 0x562c22288440 2023-11-17T00:01:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.649+0000 7f3b5149db80 10 client.0 _ll_register_callbacks cb 0x7f3b2c005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42194/0 (socket says 172.21.15.23:42194) 2023-11-17T00:01:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 -- 172.21.15.23:0/53717304 learned_addr learned my addr 172.21.15.23:0/53717304 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b377fe700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b37fff700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22288440 0x562c222964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 -- 172.21.15.23:0/53717304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22288440 msgr2=0x562c222964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22288440 0x562c222964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 -- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 msgr2=0x562c222959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 -- 172.21.15.23:0/53717304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c2219a130 con 0x562c22287100 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b377fe700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b36ffd700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c013ff0 tx=0x7f3b2c021c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c3ffaeb881de45 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.650+0000 7f3b16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0a3520 con 0x562c22287100 2023-11-17T00:01:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3b2c09b3c0 con 0x562c22287100 2023-11-17T00:01:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b28003680 con 0x562c22287100 2023-11-17T00:01:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b2c09b580 con 0x562c22287100 2023-11-17T00:01:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 10 client.15351 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562c21feac00 con 0x562c22287100 2023-11-17T00:01:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b28003680 con 0x562c22287100 2023-11-17T00:01:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3b2c0a3090 con 0x562c22287100 2023-11-17T00:01:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 10 client.15351 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 10 client.15351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.651+0000 7f3b5149db80 20 client.15351 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.652+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c222f0640 0x562c222f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.652+0000 7f3b5149db80 4 client.15351 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:01:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.652+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562c221a2600 con 0x562c222f0640 2023-11-17T00:01:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.652+0000 7f3b377fe700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c222f0640 0x562c222f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.653+0000 7f3b16ffd700 10 client.15351 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.653+0000 7f3b377fe700 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c222f0640 0x562c222f2a50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0bafc39652169a8 server_cookie=3b14b74aadc91c95 in_seq=0 out_seq=0 2023-11-17T00:01:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b16ffd700 1 -- 172.21.15.23:0/53717304 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562c221a2600 con 0x562c222f0640 2023-11-17T00:01:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b16ffd700 10 client.15351 handle_command_reply: tid=0 2023-11-17T00:01:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 client.15351 shutdown 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 msgr2=0x562c22295f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c013ff0 tx=0x7f3b2c021c40 comp rx=0 tx=0).stop 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 shutdown_connections 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c22288440 0x562c222964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c220de6b0 0x562c222959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x562c222f0640 0x562c222f2a50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 --2- 172.21.15.23:0/53717304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c22287100 0x562c22295f00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b16ffd700 0 client.15351 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.666+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 >> 172.21.15.23:0/53717304 conn(0x562c222781a0 msgr2=0x562c22290090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 shutdown_connections 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b5149db80 1 -- 172.21.15.23:0/53717304 wait complete. 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b5149db80 20 client.15351 trim_cache size 0 max 16384 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:01:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:01:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:01:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:01:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 msgr2=0x7f3b200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 0x7f3b200a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009460 tx=0x7f3b30009920 comp rx=0 tx=0).stop 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 msgr2=0x7f3b381963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.667+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 0x7f3b381963a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0098e0 tx=0x7f3b2c009580 comp rx=0 tx=0).stop 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 shutdown_connections 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b38105db0 0x7f3b3819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b38100b70 0x7f3b38195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3b200a1150 0x7f3b200a3600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 --2- 172.21.15.23:0/2589605919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b38103490 0x7f3b381963a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 >> 172.21.15.23:0/2589605919 conn(0x7f3b380fa530 msgr2=0x7f3b381017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 shutdown_connections 2023-11-17T00:01:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:28.668+0000 7f3b3c99d700 1 -- 172.21.15.23:0/2589605919 wait complete. 2023-11-17T00:01:31.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:29 smithi023 ceph-mon[97824]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:31.140 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:29 smithi156 ceph-mon[131105]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:31.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:29 smithi173 ceph-mon[131534]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2023-11-17T00:01:31.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:31 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-11-17T00:01:31.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2023-11-17T00:01:31.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:31 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-11-17T00:01:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2023-11-17T00:01:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:31 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-11-17T00:01:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:32 smithi023 ceph-mon[97824]: pgmap v380: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:01:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2023-11-17T00:01:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:32 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-11-17T00:01:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:32 smithi023 ceph-mon[97824]: osdmap e91: 12 total, 12 up, 12 in 2023-11-17T00:01:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:32 smithi023 ceph-mon[97824]: osdmap e92: 12 total, 12 up, 12 in 2023-11-17T00:01:32.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:32 smithi173 ceph-mon[131534]: pgmap v380: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:32 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:32 smithi173 ceph-mon[131534]: osdmap e91: 12 total, 12 up, 12 in 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:32 smithi173 ceph-mon[131534]: osdmap e92: 12 total, 12 up, 12 in 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:32 smithi156 ceph-mon[131105]: pgmap v380: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:01:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2023-11-17T00:01:32.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:32 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-11-17T00:01:32.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:32 smithi156 ceph-mon[131105]: osdmap e91: 12 total, 12 up, 12 in 2023-11-17T00:01:32.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:32 smithi156 ceph-mon[131105]: osdmap e92: 12 total, 12 up, 12 in 2023-11-17T00:01:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:34 smithi023 ceph-mon[97824]: pgmap v383: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2023-11-17T00:01:34.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:34 smithi173 ceph-mon[131534]: pgmap v383: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2023-11-17T00:01:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:34 smithi156 ceph-mon[131105]: pgmap v383: 129 pgs: 129 active+clean; 655 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2023-11-17T00:01:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:36 smithi023 ceph-mon[97824]: pgmap v384: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-11-17T00:01:36.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:36 smithi173 ceph-mon[131534]: pgmap v384: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-11-17T00:01:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:36 smithi156 ceph-mon[131105]: pgmap v384: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-11-17T00:01:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:38 smithi023 ceph-mon[97824]: pgmap v385: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-11-17T00:01:38.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:38 smithi173 ceph-mon[131534]: pgmap v385: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-11-17T00:01:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:38 smithi156 ceph-mon[131105]: pgmap v385: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-11-17T00:01:40.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:39 smithi023 ceph-mon[97824]: pgmap v386: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 895 B/s wr, 8 op/s 2023-11-17T00:01:40.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:39 smithi173 ceph-mon[131534]: pgmap v386: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 895 B/s wr, 8 op/s 2023-11-17T00:01:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:39 smithi156 ceph-mon[131105]: pgmap v386: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 895 B/s wr, 8 op/s 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/2945891372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec100060 msgr2=0x7fe3ec100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/2945891372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec100060 0x7fe3ec100470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009990 tx=0x7fe3e8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/2945891372 shutdown_connections 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/2945891372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/2945891372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec100060 0x7fe3ec100470 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/2945891372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec101250 0x7fe3ec1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/2945891372 >> 172.21.15.23:0/2945891372 conn(0x7fe3ec0fb640 msgr2=0x7fe3ec0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/2945891372 shutdown_connections 2023-11-17T00:01:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.908+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/2945891372 wait complete. 2023-11-17T00:01:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.910+0000 7fe3f2d9d700 1 Processor -- start 2023-11-17T00:01:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.910+0000 7fe3f2d9d700 1 -- start start 2023-11-17T00:01:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 0x7fe3ec193b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1940b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec06a510 con 0x7fe3ec101250 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec06a680 con 0x7fe3ec100060 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f2d9d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec06a840 con 0x7fe3ec101be0 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39082/0 (socket says 172.21.15.23:39082) 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f159a700 1 -- 172.21.15.23:0/3678970151 learned_addr learned my addr 172.21.15.23:0/3678970151 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 0x7fe3ec193b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.911+0000 7fe3f259c700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1940b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3678970151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 msgr2=0x7fe3ec1940b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1940b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 msgr2=0x7fe3ec199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3678970151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e80095f0 con 0x7fe3ec100060 2023-11-17T00:01:41.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f159a700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:41.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f259c700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1940b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:41.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 0x7fe3ec193b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe3e80098e0 tx=0x7fe3e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3e2ffd700 1 -- 172.21.15.23:0/3678970151 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e801d070 con 0x7fe3ec100060 2023-11-17T00:01:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.912+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec10b9c0 con 0x7fe3ec100060 2023-11-17T00:01:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.913+0000 7fe3e2ffd700 1 -- 172.21.15.23:0/3678970151 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e80054e0 con 0x7fe3ec100060 2023-11-17T00:01:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.913+0000 7fe3e2ffd700 1 -- 172.21.15.23:0/3678970151 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8004bd0 con 0x7fe3ec100060 2023-11-17T00:01:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.913+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec10b830 con 0x7fe3ec100060 2023-11-17T00:01:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.918+0000 7fe3e2ffd700 1 -- 172.21.15.23:0/3678970151 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe3e800fb90 con 0x7fe3ec100060 2023-11-17T00:01:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.919+0000 7fe4078f1b80 1 Processor -- start 2023-11-17T00:01:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.920+0000 7fe4078f1b80 1 -- start start 2023-11-17T00:01:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.920+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c27820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c27d60 0x557102c28200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102b2c000 con 0x557102a826b0 2023-11-17T00:01:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102b7e840 con 0x557102c26fb0 2023-11-17T00:01:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102b77cd0 con 0x557102c27d60 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3e2ffd700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 0x7fe3d80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39098/0 (socket says 172.21.15.23:39098) 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3f259c700 1 -- 172.21.15.23:0/3217828872 learned_addr learned my addr 172.21.15.23:0/3217828872 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3e2ffd700 1 -- 172.21.15.23:0/3678970151 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fe3e80cde50 con 0x7fe3ec100060 2023-11-17T00:01:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c27820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.921+0000 7fe3f159a700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c27d60 0x557102c28200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3217828872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c27d60 msgr2=0x557102c28200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c27d60 0x557102c28200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3217828872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 msgr2=0x557102c26a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 -- 172.21.15.23:0/3217828872 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571029f95d0 con 0x557102c26fb0 2023-11-17T00:01:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f259c700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c27820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800b2e0 tx=0x7fe3e8009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7734307a29e7ff6 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.922+0000 7fe3d7fff700 1 -- 172.21.15.23:0/3217828872 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80a3300 con 0x557102c26fb0 2023-11-17T00:01:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.923+0000 7fe3d7fff700 1 -- 172.21.15.23:0/3217828872 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e8005650 con 0x557102c26fb0 2023-11-17T00:01:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.923+0000 7fe3d7fff700 1 -- 172.21.15.23:0/3217828872 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e809a430 con 0x557102c26fb0 2023-11-17T00:01:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.923+0000 7fe3f259c700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 0x7fe3d80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.923+0000 7fe4078f1b80 1 -- 172.21.15.23:0/3217828872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 msgr2=0x557102c27820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.923+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c27820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800b2e0 tx=0x7fe3e8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe3f259c700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 0x7fe3d80a3680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x557102c21ee0 tx=0x7fe3e400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 -- 172.21.15.23:0/3217828872 shutdown_connections 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c27d60 0x557102c28200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c26a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/3217828872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c27820 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 -- 172.21.15.23:0/3217828872 >> 172.21.15.23:0/3217828872 conn(0x557102c1c1a0 msgr2=0x557102c212e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 -- 172.21.15.23:0/3217828872 shutdown_connections 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.924+0000 7fe4078f1b80 1 -- 172.21.15.23:0/3217828872 wait complete. 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.925+0000 7fe4078f1b80 1 -- start start 2023-11-17T00:01:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.925+0000 7fe4078f1b80 1 Processor -- start 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 0x557102c370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102b19880 con 0x557102c27d60 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102b19520 con 0x557102c26fb0 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571029fb8c0 con 0x557102a826b0 2023-11-17T00:01:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.926+0000 7fe4078f1b80 10 client.0 _ll_register_callbacks cb 0x557102b37720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 0x557102c370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 0x557102c370e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39106/0 (socket says 172.21.15.23:39106) 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f259c700 1 -- 172.21.15.23:0/134021029 learned_addr learned my addr 172.21.15.23:0/134021029 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 -- 172.21.15.23:0/134021029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 msgr2=0x557102c2fb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2fb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 -- 172.21.15.23:0/134021029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 msgr2=0x557102c370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 0x557102c370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.927+0000 7fe3f159a700 1 -- 172.21.15.23:0/134021029 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec10b830 con 0x557102c26fb0 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.928+0000 7fe3f159a700 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c300a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc011ce0 tx=0x7fe3dc009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ff56f9b9517cf2 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.928+0000 7fe3d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.928+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc016070 con 0x557102c26fb0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.928+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557102c19960 con 0x557102c26fb0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.928+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fe3dc00fc20 con 0x557102c26fb0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3dc00fde0 con 0x557102c26fb0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe4078f1b80 10 client.25376 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557102b3e130 con 0x557102c26fb0 2023-11-17T00:01:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557102c19960 con 0x557102c26fb0 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe3dc003cf0 con 0x557102c26fb0 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.929+0000 7fe4078f1b80 10 client.25376 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe4078f1b80 10 client.25376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe4078f1b80 20 client.25376 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c3b9e0 0x557102c3ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe4078f1b80 4 client.25376 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557102b3a040 con 0x557102c3b9e0 2023-11-17T00:01:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c3b9e0 0x557102c3ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.930+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c3b9e0 0x557102c3ddf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21654dfe7e6f335 server_cookie=73aec047dcea2bb8 in_seq=0 out_seq=0 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.931+0000 7fe3d57fa700 10 client.25376 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.931+0000 7fe3d57fa700 1 -- 172.21.15.23:0/134021029 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557102b3a040 con 0x557102c3b9e0 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.931+0000 7fe3d57fa700 10 client.25376 handle_command_reply: tid=0 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.931+0000 7fe4078f1b80 1 client.25376 shutdown 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 msgr2=0x557102c300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c300a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc011ce0 tx=0x7fe3dc009ed0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 shutdown_connections 2023-11-17T00:01:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c3b9e0 0x557102c3ddf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c27d60 0x557102c370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/134021029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c26fb0 0x557102c300a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 >> 172.21.15.23:0/134021029 conn(0x557102c1c1a0 msgr2=0x557102c20a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe3d57fa700 0 client.25376 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 shutdown_connections 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.932+0000 7fe4078f1b80 1 -- 172.21.15.23:0/134021029 wait complete. 2023-11-17T00:01:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.933+0000 7fe4078f1b80 20 client.25376 trim_cache size 0 max 16384 2023-11-17T00:01:41.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.957+0000 7fe4078f1b80 1 Processor -- start 2023-11-17T00:01:41.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.957+0000 7fe4078f1b80 1 -- start start 2023-11-17T00:01:41.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.957+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c2fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c2f010 con 0x557102c26fb0 2023-11-17T00:01:41.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c1bd70 con 0x557102c27d60 2023-11-17T00:01:41.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c27640 con 0x557102a826b0 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39108/0 (socket says 172.21.15.23:39108) 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 -- 172.21.15.23:0/2250120143 learned_addr learned my addr 172.21.15.23:0/2250120143 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f259c700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c2fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 -- 172.21.15.23:0/2250120143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 msgr2=0x557102c2f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 -- 172.21.15.23:0/2250120143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 msgr2=0x557102c2fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.958+0000 7fe3f159a700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c2fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3f159a700 1 -- 172.21.15.23:0/2250120143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557102b3de80 con 0x557102c26fb0 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3f159a700 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009d20 tx=0x7fe3dc00f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82e440b4a3405e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3e0ff9700 1 -- 172.21.15.23:0/2250120143 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc016070 con 0x557102c26fb0 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3e0ff9700 1 -- 172.21.15.23:0/2250120143 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3dc01a770 con 0x557102c26fb0 2023-11-17T00:01:41.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.959+0000 7fe3e0ff9700 1 -- 172.21.15.23:0/2250120143 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc005660 con 0x557102c26fb0 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 -- 172.21.15.23:0/2250120143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 msgr2=0x557102c37000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009d20 tx=0x7fe3dc00f6a0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 -- 172.21.15.23:0/2250120143 shutdown_connections 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c2fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102a826b0 0x557102c2f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/2250120143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102c26fb0 0x557102c37000 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 -- 172.21.15.23:0/2250120143 >> 172.21.15.23:0/2250120143 conn(0x557102c1c1a0 msgr2=0x557102c2aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 -- 172.21.15.23:0/2250120143 shutdown_connections 2023-11-17T00:01:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.960+0000 7fe4078f1b80 1 -- 172.21.15.23:0/2250120143 wait complete. 2023-11-17T00:01:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.961+0000 7fe4078f1b80 1 -- start start 2023-11-17T00:01:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.961+0000 7fe4078f1b80 1 Processor -- start 2023-11-17T00:01:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c35240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c35780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c36890 con 0x557102a826b0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c36a00 con 0x557102c27d60 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557102c36bc0 con 0x557102c35cc0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe4078f1b80 10 client.0 _ll_register_callbacks cb 0x7fe3dc0117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe3f159a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c35780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe3f259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:58634/0 (socket says 172.21.15.23:58634) 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.962+0000 7fe3f259c700 1 -- 172.21.15.23:0/253056469 learned_addr learned my addr 172.21.15.23:0/253056469 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c35240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 -- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 msgr2=0x557102c35780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c35780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 -- 172.21.15.23:0/253056469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 msgr2=0x557102c35240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c35240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 -- 172.21.15.23:0/253056469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571029f95d0 con 0x557102c35cc0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f159a700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c35780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c35240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3f259c700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4013cc0 tx=0x7fe3e4009130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1d283b1804098fd server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.963+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e401c040 con 0x557102c35cc0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.964+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fe3e40046c0 con 0x557102c35cc0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.964+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3e8005650 con 0x557102c35cc0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.964+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3e4003e50 con 0x557102c35cc0 2023-11-17T00:01:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.964+0000 7fe4078f1b80 10 client.25420 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.964+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557102b3e130 con 0x557102c35cc0 2023-11-17T00:01:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3e8005650 con 0x557102c35cc0 2023-11-17T00:01:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe3e4026420 con 0x557102c35cc0 2023-11-17T00:01:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 10 client.25420 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 10 client.25420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 20 client.25420 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c909b0 0x557102c92dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 4 client.25420 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557102b46600 con 0x557102c909b0 2023-11-17T00:01:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.965+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c909b0 0x557102c92dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.966+0000 7fe3f1d9b700 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c909b0 0x557102c92dc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2209badad198c8e server_cookie=d00148c5489f7f71 in_seq=0 out_seq=0 2023-11-17T00:01:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.966+0000 7fe3d57fa700 10 client.25420 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.973+0000 7fe3d57fa700 1 -- 172.21.15.23:0/253056469 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557102b46600 con 0x557102c909b0 2023-11-17T00:01:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.973+0000 7fe3d57fa700 10 client.25420 handle_command_reply: tid=0 2023-11-17T00:01:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.973+0000 7fe4078f1b80 1 client.25420 shutdown 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 msgr2=0x557102c36130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4013cc0 tx=0x7fe3e4009130 comp rx=0 tx=0).stop 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 shutdown_connections 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557102c35cc0 0x557102c36130 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557102c909b0 0x557102c92dc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557102a826b0 0x557102c35240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 --2- 172.21.15.23:0/253056469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557102c27d60 0x557102c35780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 >> 172.21.15.23:0/253056469 conn(0x557102c1c1a0 msgr2=0x557102c2f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe3d57fa700 0 client.25420 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 shutdown_connections 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 1 -- 172.21.15.23:0/253056469 wait complete. 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.974+0000 7fe4078f1b80 20 client.25420 trim_cache size 0 max 16384 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:01:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:01:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 msgr2=0x7fe3d80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 0x7fe3d80a3680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x557102c21ee0 tx=0x7fe3e400b040 comp rx=0 tx=0).stop 2023-11-17T00:01:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 msgr2=0x7fe3ec193b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 0x7fe3ec193b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe3e80098e0 tx=0x7fe3e8004730 comp rx=0 tx=0).stop 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 shutdown_connections 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fe3d80a11d0 0x7fe3d80a3680 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3ec101be0 0x7fe3ec1940b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100060 0x7fe3ec193b70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 --2- 172.21.15.23:0/3678970151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3ec101250 0x7fe3ec199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.975+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 >> 172.21.15.23:0/3678970151 conn(0x7fe3ec0fb640 msgr2=0x7fe3ec0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.976+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 shutdown_connections 2023-11-17T00:01:41.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:41.976+0000 7fe3f2d9d700 1 -- 172.21.15.23:0/3678970151 wait complete. 2023-11-17T00:01:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:42 smithi023 ceph-mon[97824]: pgmap v387: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 718 B/s wr, 8 op/s 2023-11-17T00:01:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:42 smithi173 ceph-mon[131534]: pgmap v387: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 718 B/s wr, 8 op/s 2023-11-17T00:01:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:42 smithi156 ceph-mon[131105]: pgmap v387: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 718 B/s wr, 8 op/s 2023-11-17T00:01:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:44 smithi023 ceph-mon[97824]: pgmap v388: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 6 op/s 2023-11-17T00:01:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:44 smithi156 ceph-mon[131105]: pgmap v388: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 6 op/s 2023-11-17T00:01:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:44 smithi173 ceph-mon[131534]: pgmap v388: 129 pgs: 129 active+clean; 655 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 186 B/s wr, 6 op/s 2023-11-17T00:01:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:46 smithi023 ceph-mon[97824]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-17T00:01:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:46 smithi156 ceph-mon[131105]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-17T00:01:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:46 smithi173 ceph-mon[131534]: pgmap v389: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-17T00:01:48.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:48 smithi023 ceph-mon[97824]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:48.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:48 smithi156 ceph-mon[131105]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:48.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:48 smithi173 ceph-mon[131534]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:49 smithi023 ceph-mon[97824]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:49 smithi173 ceph-mon[131534]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:50.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:49 smithi156 ceph-mon[131105]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:01:52.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:52 smithi023 ceph-mon[97824]: pgmap v392: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-11-17T00:01:52.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:52 smithi173 ceph-mon[131534]: pgmap v392: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-11-17T00:01:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:52 smithi156 ceph-mon[131105]: pgmap v392: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-11-17T00:01:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:54 smithi023 ceph-mon[97824]: pgmap v393: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:57.217 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:54 smithi156 ceph-mon[131105]: pgmap v393: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:57.217 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:56 smithi156 ceph-mon[131105]: pgmap v394: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:57.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:54 smithi173 ceph-mon[131534]: pgmap v393: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-11-17T00:01:57.218 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:56 smithi173 ceph-mon[131534]: pgmap v394: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 -- 172.21.15.23:0/1678727040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 msgr2=0x7f0824108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 --2- 172.21.15.23:0/1678727040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f0824108700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f081c009f90 tx=0x7f081c00b330 comp rx=0 tx=0).stop 2023-11-17T00:01:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 -- 172.21.15.23:0/1678727040 shutdown_connections 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 --2- 172.21.15.23:0/1678727040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f0824108700 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 --2- 172.21.15.23:0/1678727040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241002c0 0x7f08241006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 --2- 172.21.15.23:0/1678727040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241014b0 0x7f0824101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 -- 172.21.15.23:0/1678727040 >> 172.21.15.23:0/1678727040 conn(0x7f08240fb8e0 msgr2=0x7f08240fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 -- 172.21.15.23:0/1678727040 shutdown_connections 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.129+0000 7f0828dff700 1 -- 172.21.15.23:0/1678727040 wait complete. 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.130+0000 7f0828dff700 1 Processor -- start 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.130+0000 7f0828dff700 1 -- start start 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241002c0 0x7f08241982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 0x7f0824198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419fe70 con 0x7f0824101e40 2023-11-17T00:01:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419ffb0 con 0x7f08241002c0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0828dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08241a0140 con 0x7f08241014b0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0822d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f082259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241002c0 0x7f08241982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 0x7f0824198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0822d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36696/0 (socket says 172.21.15.23:36696) 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0822d9d700 1 -- 172.21.15.23:0/913762436 learned_addr learned my addr 172.21.15.23:0/913762436 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 -- 172.21.15.23:0/913762436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241002c0 msgr2=0x7f08241982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241002c0 0x7f08241982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 -- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 msgr2=0x7f082419f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0821d9b700 1 -- 172.21.15.23:0/913762436 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c009c40 con 0x7f08241014b0 2023-11-17T00:01:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.131+0000 7f0822d9d700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f0821d9b700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 0x7f0824198830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f08137fe700 1 -- 172.21.15.23:0/913762436 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081800c9c0 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f08137fe700 1 -- 172.21.15.23:0/913762436 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0818010430 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082410bc30 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f08137fe700 1 -- 172.21.15.23:0/913762436 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818018680 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.132+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082410baa0 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.136+0000 7f08137fe700 1 -- 172.21.15.23:0/913762436 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0818010860 con 0x7f08241014b0 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 Processor -- start 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 -- start start 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc33870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdb39000 con 0x5600bdc33db0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdb8b840 con 0x5600bdc34b10 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdb84cd0 con 0x5600bda8f6b0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.137+0000 7f08137fe700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 0x7f080c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f082259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f082259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36706/0 (socket says 172.21.15.23:36706) 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f082259c700 1 -- 172.21.15.23:0/593573337 learned_addr learned my addr 172.21.15.23:0/593573337 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0821d9b700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc33870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 0x7f080c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f08137fe700 1 -- 172.21.15.23:0/913762436 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0818014070 con 0x7f08241014b0 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 -- 172.21.15.23:0/593573337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 msgr2=0x5600bdc34f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc34f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 -- 172.21.15.23:0/593573337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 msgr2=0x5600bdc345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f0822d9d700 1 -- 172.21.15.23:0/593573337 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bda065d0 con 0x5600bda8f6b0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.138+0000 7f082259c700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.139+0000 7f0822d9d700 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc33870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f081c007930 tx=0x7f081c008e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60f9f7f9f2280656 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.139+0000 7f0822d9d700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 0x7f080c0a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f081c01a000 tx=0x7f081c026000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.139+0000 7f0810ff9700 1 -- 172.21.15.23:0/593573337 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c00e040 con 0x5600bda8f6b0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.139+0000 7f0810ff9700 1 -- 172.21.15.23:0/593573337 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f081c00be50 con 0x5600bda8f6b0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.139+0000 7f0810ff9700 1 -- 172.21.15.23:0/593573337 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c01ead0 con 0x5600bda8f6b0 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- 172.21.15.23:0/593573337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 msgr2=0x5600bdc33870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc33870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f081c007930 tx=0x7f081c008e20 comp rx=0 tx=0).stop 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- 172.21.15.23:0/593573337 shutdown_connections 2023-11-17T00:01:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc33870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/593573337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- 172.21.15.23:0/593573337 >> 172.21.15.23:0/593573337 conn(0x5600bdc28f50 msgr2=0x5600bdc2e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- 172.21.15.23:0/593573337 shutdown_connections 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- 172.21.15.23:0/593573337 wait complete. 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.140+0000 7f083c7d4b80 1 -- start start 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 Processor -- start 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc3cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc44220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdb26880 con 0x5600bdc33db0 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdb26520 con 0x5600bdc34b10 2023-11-17T00:01:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bda088c0 con 0x5600bda8f6b0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f083c7d4b80 10 client.0 _ll_register_callbacks cb 0x5600bdb44720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36712/0 (socket says 172.21.15.23:36712) 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f0821d9b700 1 -- 172.21.15.23:0/2086906422 learned_addr learned my addr 172.21.15.23:0/2086906422 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.141+0000 7f0822d9d700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc44220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f082259c700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc3cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 -- 172.21.15.23:0/2086906422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 msgr2=0x5600bdc3cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc3cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 -- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 msgr2=0x5600bdc44220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc44220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 -- 172.21.15.23:0/2086906422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0824109030 con 0x5600bdc33db0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0821d9b700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0818099fc0 tx=0x7f0818065970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b50461c591c553 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0809ffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081809a030 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f081809e060 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.142+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600bdc26710 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08180a2480 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 10 client.15378 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5600bdb4b130 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5600bdc26710 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f08180a1ff0 con 0x5600bdc33db0 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 10 client.15378 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 10 client.15378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 20 client.15378 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc48ad0 0x5600bdc4aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 4 client.15378 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600bdb47040 con 0x5600bdc48ad0 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.143+0000 7f082259c700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc48ad0 0x5600bdc4aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.144+0000 7f082259c700 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc48ad0 0x5600bdc4aee0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87faaece15ec4892 server_cookie=f343b1acf580aeac in_seq=0 out_seq=0 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.144+0000 7f0809ffb700 10 client.15378 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f0809ffb700 1 -- 172.21.15.23:0/2086906422 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5600bdb47040 con 0x5600bdc48ad0 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f0809ffb700 10 client.15378 handle_command_reply: tid=0 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 client.15378 shutdown 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 msgr2=0x5600bdc3d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0818099fc0 tx=0x7f0818065970 comp rx=0 tx=0).stop 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 shutdown_connections 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc44220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bda8f6b0 0x5600bdc3cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc48ad0 0x5600bdc4aee0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/2086906422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3d1e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 >> 172.21.15.23:0/2086906422 conn(0x5600bdc28f50 msgr2=0x5600bdc2d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f0809ffb700 0 client.15378 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 shutdown_connections 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 1 -- 172.21.15.23:0/2086906422 wait complete. 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.145+0000 7f083c7d4b80 20 client.15378 trim_cache size 0 max 16384 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 Processor -- start 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 -- start start 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bda8f6b0 0x5600bdc3b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc3aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc3c150 con 0x5600bda8f6b0 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc3c810 con 0x5600bdc34b10 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc2fb90 con 0x5600bdc33db0 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f0822d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc3aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.161+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0822d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc3aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34118/0 (socket says 172.21.15.23:34118) 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f082259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bda8f6b0 0x5600bdc3b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:34270/0 (socket says 172.21.15.23:34270) 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0822d9d700 1 -- 172.21.15.23:0/1844650309 learned_addr learned my addr 172.21.15.23:0/1844650309 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 -- 172.21.15.23:0/1844650309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 msgr2=0x5600bdc3aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc3aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 -- 172.21.15.23:0/1844650309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bda8f6b0 msgr2=0x5600bdc3b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bda8f6b0 0x5600bdc3b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 -- 172.21.15.23:0/1844650309 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bdb4ae80 con 0x5600bdc33db0 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.162+0000 7f0821d9b700 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9c0 tx=0x7f0818010c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18df399feb03e36a server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.165+0000 7f08117fa700 1 -- 172.21.15.23:0/1844650309 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818065e70 con 0x5600bdc33db0 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.165+0000 7f08117fa700 1 -- 172.21.15.23:0/1844650309 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f081809e060 con 0x5600bdc33db0 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.165+0000 7f08117fa700 1 -- 172.21.15.23:0/1844650309 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818099520 con 0x5600bdc33db0 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 -- 172.21.15.23:0/1844650309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 msgr2=0x5600bdc441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9c0 tx=0x7f0818010c60 comp rx=0 tx=0).stop 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 -- 172.21.15.23:0/1844650309 shutdown_connections 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bdc34b10 0x5600bdc3aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bda8f6b0 0x5600bdc3b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/1844650309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc33db0 0x5600bdc441e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 -- 172.21.15.23:0/1844650309 >> 172.21.15.23:0/1844650309 conn(0x5600bdc28f50 msgr2=0x5600bdc4f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 -- 172.21.15.23:0/1844650309 shutdown_connections 2023-11-17T00:01:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.166+0000 7f083c7d4b80 1 -- 172.21.15.23:0/1844650309 wait complete. 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 -- start start 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 Processor -- start 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc34b10 0x5600bdc42c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc43170 con 0x5600bdc33db0 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc432e0 con 0x5600bda8f6b0 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bdc434a0 con 0x5600bdc34b10 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.167+0000 7f083c7d4b80 10 client.0 _ll_register_callbacks cb 0x7f0818099e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f0821d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34134/0 (socket says 172.21.15.23:34134) 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 -- 172.21.15.23:0/3249270276 learned_addr learned my addr 172.21.15.23:0/3249270276 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f0822d9d700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc34b10 0x5600bdc42c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 -- 172.21.15.23:0/3249270276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc34b10 msgr2=0x5600bdc42c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc34b10 0x5600bdc42c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 -- 172.21.15.23:0/3249270276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 msgr2=0x5600bdc3fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 -- 172.21.15.23:0/3249270276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600bdb4b130 con 0x5600bda8f6b0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f0821d9b700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f082259c700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5600bdc4b7c0 tx=0x7f081401f680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aef7ce4b6245b03 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.168+0000 7f0809ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814004220 con 0x5600bda8f6b0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0814004380 con 0x5600bda8f6b0 2023-11-17T00:01:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f081c00be50 con 0x5600bda8f6b0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0814003680 con 0x5600bda8f6b0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 10 client.25400 fetch_fsmap learned FSMap version 24 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5600bd99bc00 con 0x5600bda8f6b0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f081c00be50 con 0x5600bda8f6b0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0814010a40 con 0x5600bda8f6b0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 10 client.25400 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 10 client.25400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.169+0000 7f083c7d4b80 20 client.25400 populate_metadata read hostname 'smithi023' 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc9d130 0x5600bdc9f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f083c7d4b80 4 client.25400 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5600bdb53600 con 0x5600bdc9d130 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f0821d9b700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc9d130 0x5600bdc9f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:01:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f0821d9b700 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc9d130 0x5600bdc9f540 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cef6cdbe9ea35959 server_cookie=de2ee6d639b1f1e6 in_seq=0 out_seq=0 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.170+0000 7f0809ffb700 10 client.25400 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f0809ffb700 1 -- 172.21.15.23:0/3249270276 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5600bdb53600 con 0x5600bdc9d130 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f0809ffb700 10 client.25400 handle_command_reply: tid=0 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 client.25400 shutdown 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 msgr2=0x5600bdc3f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5600bdc4b7c0 tx=0x7f081401f680 comp rx=0 tx=0).stop 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 shutdown_connections 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600bdc34b10 0x5600bdc42c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600bda8f6b0 0x5600bdc3f860 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5600bdc9d130 0x5600bdc9f540 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 --2- 172.21.15.23:0/3249270276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600bdc33db0 0x5600bdc3fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f0809ffb700 0 client.25400 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:01:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.178+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 >> 172.21.15.23:0/3249270276 conn(0x5600bdc28f50 msgr2=0x5600bdc3b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 shutdown_connections 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f083c7d4b80 1 -- 172.21.15.23:0/3249270276 wait complete. 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f083c7d4b80 20 client.25400 trim_cache size 0 max 16384 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:01:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 msgr2=0x7f080c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 0x7f080c0a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f081c01a000 tx=0x7f081c026000 comp rx=0 tx=0).stop 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 msgr2=0x7f0824198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 0x7f0824198830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800ed00 comp rx=0 tx=0).stop 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 shutdown_connections 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0824101e40 0x7f082419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08241002c0 0x7f08241982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f080c0a1150 0x7f080c0a3600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 --2- 172.21.15.23:0/913762436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08241014b0 0x7f0824198830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:01:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.179+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 >> 172.21.15.23:0/913762436 conn(0x7f08240fb8e0 msgr2=0x7f08240fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:01:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.180+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 shutdown_connections 2023-11-17T00:01:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:01:55.180+0000 7f0828dff700 1 -- 172.21.15.23:0/913762436 wait complete. 2023-11-17T00:01:57.251 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 67% (8133/12072) Updating files: 68% (8209/12072) Updating files: 69% (8330/12072) Updating files: 70% (8451/12072) Updating files: 71% (8572/12072) Updating files: 72% (8692/12072) Updating files: 73% (8813/12072) Updating files: 74% (8934/12072) Updating files: 75% (9054/12072) Updating files: 76% (9175/12072) Updating files: 77% (9296/12072) Updating files: 78% (9417/12072) Updating files: 79% (9537/12072) Updating files: 80% (9658/12072) Updating files: 81% (9779/12072) Updating files: 82% (9900/12072) Updating files: 83% (10020/12072) Updating files: 84% (10141/12072) Updating files: 85% (10262/12072) Updating files: 86% (10382/12072) Updating files: 87% (10503/12072) Updating files: 88% (10624/12072) Updating files: 89% (10745/12072) Updating files: 90% (10865/12072) Updating files: 91% (10986/12072) Updating files: 92% (11107/12072) Updating files: 93% (11227/12072) Updating files: 94% (11348/12072) Updating files: 95% (11469/12072) Updating files: 96% (11590/12072) Updating files: 97% (11710/12072) Updating files: 98% (11831/12072) Updating files: 99% (11952/12072) Updating files: 100% (12072/12072) Updating files: 100% (12072/12072), done. 2023-11-17T00:01:57.251 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:56 smithi023 ceph-mon[97824]: pgmap v394: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-11-17T00:01:57.328 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to '1c9565146c8647d4e00eb6c7ae626dc192680e7b'. 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-17T00:01:57.406 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-11-17T00:01:57.407 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at 1c9565146c8 Other1 2023-11-17T00:01:57.408 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-17T00:01:57.509 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-17T00:01:57.514 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-17T00:01:57.514 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-17T00:01:58.063 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-17T00:01:58.154 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-17T00:01:58.220 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-17T00:01:58.224 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-17T00:01:58.225 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-17T00:01:58.291 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-17T00:01:58.299 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-17T00:01:58.299 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-17T00:01:58.343 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-11-17T00:01:58.345 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-11-17T00:01:58.345 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-11-17T00:01:58.440 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-11-17T00:01:58.440 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p testdir 2023-11-17T00:01:58.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd testdir 2023-11-17T00:01:58.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ set +e 2023-11-17T00:01:58.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfacl -d -m u:nobody:rw . 2023-11-17T00:01:58.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:01:58.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-11-17T00:01:58.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=0 2023-11-17T00:01:58.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 0 -lt 100 ']' 2023-11-17T00:01:58.463 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 0 + 1 2023-11-17T00:01:58.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=1 2023-11-17T00:01:58.465 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:01:58.470 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:01:58.470 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:01:58.478 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:01:58.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:01:58.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:01:58.499 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:58 smithi023 ceph-mon[97824]: pgmap v395: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:01:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:58 smithi173 ceph-mon[131534]: pgmap v395: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:01:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:58 smithi156 ceph-mon[131105]: pgmap v395: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-11-17T00:02:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:01:59 smithi023 ceph-mon[97824]: pgmap v396: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-17T00:02:00.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:01:59 smithi173 ceph-mon[131534]: pgmap v396: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-17T00:02:00.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:01:59 smithi156 ceph-mon[131105]: pgmap v396: 129 pgs: 129 active+clean; 666 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-17T00:02:01.339 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:01.339 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:01.424 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:01.424 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:01.438 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:01.438 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:01.438 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 1 -lt 100 ']' 2023-11-17T00:02:01.438 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 1 + 1 2023-11-17T00:02:01.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=2 2023-11-17T00:02:01.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:01.457 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:01.458 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:01.461 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:01.461 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:01.461 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:01.858 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:01.859 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:01.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:01.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:01.958 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:01.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:01.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -lt 100 ']' 2023-11-17T00:02:01.959 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 2 + 1 2023-11-17T00:02:01.988 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=3 2023-11-17T00:02:01.988 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:02.002 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:02.002 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:02.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:02.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:02.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:02.222 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:02.222 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:02.327 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:02.327 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:02.336 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:02.337 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:02.337 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 3 -lt 100 ']' 2023-11-17T00:02:02.337 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 3 + 1 2023-11-17T00:02:02.371 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=4 2023-11-17T00:02:02.371 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:02.399 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:02.400 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:02.402 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:02.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:02.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:02 smithi023 ceph-mon[97824]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-11-17T00:02:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:02 smithi173 ceph-mon[131534]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-11-17T00:02:02.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:02 smithi156 ceph-mon[131105]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-11-17T00:02:02.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:02.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:02.844 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:02.844 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:02.855 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:02.855 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:02.855 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 4 -lt 100 ']' 2023-11-17T00:02:02.856 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 4 + 1 2023-11-17T00:02:02.890 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=5 2023-11-17T00:02:02.890 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:02.918 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:02.918 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:02.920 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:02.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:02.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:03.280 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:03.280 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:03.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:03.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:03.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:03.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:03.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 5 -lt 100 ']' 2023-11-17T00:02:03.662 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 5 + 1 2023-11-17T00:02:03.695 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=6 2023-11-17T00:02:03.696 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:03.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:03.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:03.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:03.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:03.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:04.122 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:04.122 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:04.243 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:04.243 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:04.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:04.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:04.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 6 -lt 100 ']' 2023-11-17T00:02:04.255 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 6 + 1 2023-11-17T00:02:04.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:04 smithi023 ceph-mon[97824]: pgmap v398: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-17T00:02:04.287 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=7 2023-11-17T00:02:04.287 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:04.322 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:04.322 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:04.324 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:04.325 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:04.325 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:04.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:04 smithi173 ceph-mon[131534]: pgmap v398: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-17T00:02:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:04 smithi156 ceph-mon[131105]: pgmap v398: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-11-17T00:02:04.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:04.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:04.851 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:04.852 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:04.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:04.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:04.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 7 -lt 100 ']' 2023-11-17T00:02:04.862 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 7 + 1 2023-11-17T00:02:04.906 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=8 2023-11-17T00:02:04.907 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:04.926 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:04.926 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:04.929 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:04.929 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:04.931 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:05.305 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:05.305 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:05.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:05.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:05.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:05.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:05.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 8 -lt 100 ']' 2023-11-17T00:02:05.480 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 8 + 1 2023-11-17T00:02:05.514 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=9 2023-11-17T00:02:05.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:05.543 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:05.543 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:05.545 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:05.546 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:05.546 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:05.938 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:05.938 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:06.076 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:06.076 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:06.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:06.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:06.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 9 -lt 100 ']' 2023-11-17T00:02:06.087 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 9 + 1 2023-11-17T00:02:06.120 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=10 2023-11-17T00:02:06.120 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:06.150 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:06.150 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:06.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:06.154 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:06.154 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:06.468 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:06 smithi023 ceph-mon[97824]: pgmap v399: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:02:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:06 smithi173 ceph-mon[131534]: pgmap v399: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:02:06.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:06 smithi156 ceph-mon[131105]: pgmap v399: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-11-17T00:02:06.621 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:06.622 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:06.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:06.735 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:06.744 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:06.744 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:06.745 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 10 -lt 100 ']' 2023-11-17T00:02:06.745 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 10 + 1 2023-11-17T00:02:06.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=11 2023-11-17T00:02:06.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:06.809 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:06.809 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:06.811 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:06.812 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:06.812 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:07.246 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:07.247 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:07.368 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:07.368 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:07.378 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:07.378 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:07.378 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 11 -lt 100 ']' 2023-11-17T00:02:07.378 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 11 + 1 2023-11-17T00:02:07.414 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=12 2023-11-17T00:02:07.414 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:07.442 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:07.442 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:07.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:07.445 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:07.445 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:07.946 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:07.946 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:08.068 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:08.068 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:08.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:08.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:08.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 12 -lt 100 ']' 2023-11-17T00:02:08.079 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 12 + 1 2023-11-17T00:02:08.114 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=13 2023-11-17T00:02:08.115 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:08.143 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:08.143 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:08.145 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:08.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:08.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:08.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:08 smithi023 ceph-mon[97824]: pgmap v400: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-17T00:02:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:08 smithi156 ceph-mon[131105]: pgmap v400: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-17T00:02:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:08 smithi173 ceph-mon[131534]: pgmap v400: 129 pgs: 129 active+clean; 680 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-17T00:02:08.729 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:08.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:08.859 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:08.859 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:08.869 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:08.869 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:08.869 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 13 -lt 100 ']' 2023-11-17T00:02:08.870 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 13 + 1 2023-11-17T00:02:08.903 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=14 2023-11-17T00:02:08.904 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:08.944 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:08.944 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:08.945 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:08.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:08.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:09.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:09.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:09.984 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:09.984 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:10.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:10.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:10.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 14 -lt 100 ']' 2023-11-17T00:02:10.004 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 14 + 1 2023-11-17T00:02:10.017 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:09 smithi023 ceph-mon[97824]: pgmap v401: 129 pgs: 129 active+clean; 680 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-11-17T00:02:10.036 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=15 2023-11-17T00:02:10.037 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:10.088 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:10.088 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:10.091 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:10.091 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:10.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:10.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:09 smithi173 ceph-mon[131534]: pgmap v401: 129 pgs: 129 active+clean; 680 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-11-17T00:02:10.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:09 smithi156 ceph-mon[131105]: pgmap v401: 129 pgs: 129 active+clean; 680 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-11-17T00:02:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.241+0000 7fc55c5ff700 1 -- 172.21.15.23:0/1303489785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554104000 msgr2=0x7fc55410a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.241+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/1303489785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554104000 0x7fc55410a8c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc54c00b0a0 tx=0x7fc54c00b3b0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 -- 172.21.15.23:0/1303489785 shutdown_connections 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/1303489785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554104000 0x7fc55410a8c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/1303489785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554102480 0x7fc554102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/1303489785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554103670 0x7fc554103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 -- 172.21.15.23:0/1303489785 >> 172.21.15.23:0/1303489785 conn(0x7fc5540fdac0 msgr2=0x7fc5540ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 -- 172.21.15.23:0/1303489785 shutdown_connections 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.242+0000 7fc55c5ff700 1 -- 172.21.15.23:0/1303489785 wait complete. 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 Processor -- start 2023-11-17T00:02:10.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 -- start start 2023-11-17T00:02:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 0x7fc554198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554104000 0x7fc55419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55419fe70 con 0x7fc554102480 2023-11-17T00:02:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55419ffb0 con 0x7fc554103670 2023-11-17T00:02:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55c5ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541a0140 con 0x7fc554104000 2023-11-17T00:02:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58842/0 (socket says 172.21.15.23:58842) 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc55a39b700 1 -- 172.21.15.23:0/895084859 learned_addr learned my addr 172.21.15.23:0/895084859 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.243+0000 7fc559b9a700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 0x7fc554198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554104000 0x7fc55419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 -- 172.21.15.23:0/895084859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554104000 msgr2=0x7fc55419f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554104000 0x7fc55419f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 -- 172.21.15.23:0/895084859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 msgr2=0x7fc554198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 0x7fc554198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 -- 172.21.15.23:0/895084859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c009d00 con 0x7fc554102480 2023-11-17T00:02:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc559b9a700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 0x7fc554198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc55a39b700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e970 tx=0x7fc55000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.244+0000 7fc54b7fe700 1 -- 172.21.15.23:0/895084859 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550009790 con 0x7fc554102480 2023-11-17T00:02:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.245+0000 7fc54b7fe700 1 -- 172.21.15.23:0/895084859 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc55000c7e0 con 0x7fc554102480 2023-11-17T00:02:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.245+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55410de40 con 0x7fc554102480 2023-11-17T00:02:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.245+0000 7fc54b7fe700 1 -- 172.21.15.23:0/895084859 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550003c70 con 0x7fc554102480 2023-11-17T00:02:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.245+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55410dcb0 con 0x7fc554102480 2023-11-17T00:02:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.248+0000 7fc54b7fe700 1 -- 172.21.15.23:0/895084859 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc55000cc50 con 0x7fc554102480 2023-11-17T00:02:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.250+0000 7fc54b7fe700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 0x7fc5400a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.250+0000 7fc54b7fe700 1 -- 172.21.15.23:0/895084859 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc550014070 con 0x7fc554102480 2023-11-17T00:02:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.250+0000 7fc559b9a700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 0x7fc5400a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.250+0000 7fc559b9a700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 0x7fc5400a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc544011490 tx=0x7fc544009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.322+0000 7fc56fe3db80 1 Processor -- start 2023-11-17T00:02:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 -- start start 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a890f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a795000 con 0x55fe3a88fdb0 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a7e7840 con 0x55fe3a890b10 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a7e0cd0 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc559b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc559b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58846/0 (socket says 172.21.15.23:58846) 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc559b9a700 1 -- 172.21.15.23:0/2035237064 learned_addr learned my addr 172.21.15.23:0/2035237064 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a890f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.323+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 -- 172.21.15.23:0/2035237064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a6eb6b0 msgr2=0x55fe3a88f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 -- 172.21.15.23:0/2035237064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 msgr2=0x55fe3a890f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a890f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 -- 172.21.15.23:0/2035237064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe3a6625d0 con 0x55fe3a88fdb0 2023-11-17T00:02:10.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc5440043c0 tx=0x7fc544014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45401b83d29d8c24 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc548bfe700 1 -- 172.21.15.23:0/2035237064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544003d10 con 0x55fe3a88fdb0 2023-11-17T00:02:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc548bfe700 1 -- 172.21.15.23:0/2035237064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc544003e70 con 0x55fe3a88fdb0 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.324+0000 7fc548bfe700 1 -- 172.21.15.23:0/2035237064 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54401e810 con 0x55fe3a88fdb0 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2035237064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 msgr2=0x55fe3a8905d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc5440043c0 tx=0x7fc544014000 comp rx=0 tx=0).stop 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2035237064 shutdown_connections 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a890f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2035237064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8905d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2035237064 >> 172.21.15.23:0/2035237064 conn(0x55fe3a884f50 msgr2=0x55fe3a88a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2035237064 shutdown_connections 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.325+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2035237064 wait complete. 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 -- start start 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 Processor -- start 2023-11-17T00:02:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8991e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a8a0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a782880 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a782520 con 0x55fe3a88fdb0 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.326+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a6648c0 con 0x55fe3a890b10 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc56fe3db80 10 client.0 _ll_register_callbacks cb 0x55fe3a7a0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58854/0 (socket says 172.21.15.23:58854) 2023-11-17T00:02:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 -- 172.21.15.23:0/2475863173 learned_addr learned my addr 172.21.15.23:0/2475863173 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a8a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 -- 172.21.15.23:0/2475863173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 msgr2=0x55fe3a8a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a8a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 -- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a88fdb0 msgr2=0x55fe3a8991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 -- 172.21.15.23:0/2475863173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc544017070 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc5500ca540 tx=0x7fc550003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a12ee387963da3 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.327+0000 7fc53dffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55009a310 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc5500a2300 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe3a882710 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5500b8250 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 10 client.15396 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fe3a7a7130 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fe3a882710 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5500a1e70 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 10 client.15396 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 20 client.15396 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8a4a10 0x55fe3a8a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 4 client.15396 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.328+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe3a7a3040 con 0x55fe3a8a4a10 2023-11-17T00:02:10.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.329+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8a4a10 0x55fe3a8a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.329+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8a4a10 0x55fe3a8a6e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b696f9f3898ea35e server_cookie=97a7aa46229bb2fc in_seq=0 out_seq=0 2023-11-17T00:02:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.329+0000 7fc53dffb700 10 client.15396 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc53dffb700 1 -- 172.21.15.23:0/2475863173 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fe3a7a3040 con 0x55fe3a8a4a10 2023-11-17T00:02:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc53dffb700 10 client.15396 handle_command_reply: tid=0 2023-11-17T00:02:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 client.15396 shutdown 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 msgr2=0x55fe3a898ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc5500ca540 tx=0x7fc550003e50 comp rx=0 tx=0).stop 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 shutdown_connections 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a8a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8a4a10 0x55fe3a8a6e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a898ca0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2475863173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc53dffb700 0 client.15396 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.330+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 >> 172.21.15.23:0/2475863173 conn(0x55fe3a884f50 msgr2=0x55fe3a8897e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.331+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 shutdown_connections 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.331+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2475863173 wait complete. 2023-11-17T00:02:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.331+0000 7fc56fe3db80 20 client.15396 trim_cache size 0 max 16384 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.346+0000 7fc56fe3db80 1 Processor -- start 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.346+0000 7fc56fe3db80 1 -- start start 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a8ab780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a894530 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a88aa00 con 0x55fe3a890b10 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a8abcc0 con 0x55fe3a88fdb0 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58868/0 (socket says 172.21.15.23:58868) 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 -- 172.21.15.23:0/2649616334 learned_addr learned my addr 172.21.15.23:0/2649616334 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc559b9a700 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a8ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 -- 172.21.15.23:0/2649616334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a88fdb0 msgr2=0x55fe3a8a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 -- 172.21.15.23:0/2649616334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 msgr2=0x55fe3a8ab780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a8ab780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.347+0000 7fc55a39b700 1 -- 172.21.15.23:0/2649616334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe3a7a6e80 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc55a39b700 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc5500963b0 tx=0x7fc550098460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fe3e1f6d4a8550 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc5493ff700 1 -- 172.21.15.23:0/2649616334 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5500add50 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc5493ff700 1 -- 172.21.15.23:0/2649616334 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc550099eb0 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc5493ff700 1 -- 172.21.15.23:0/2649616334 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550099600 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2649616334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 msgr2=0x55fe3a88a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc5500963b0 tx=0x7fc550098460 comp rx=0 tx=0).stop 2023-11-17T00:02:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2649616334 shutdown_connections 2023-11-17T00:02:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a890b10 0x55fe3a8ab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a88a0d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/2649616334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a88fdb0 0x55fe3a8a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.348+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2649616334 >> 172.21.15.23:0/2649616334 conn(0x55fe3a884f50 msgr2=0x55fe3a897a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:10.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.349+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2649616334 shutdown_connections 2023-11-17T00:02:10.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.349+0000 7fc56fe3db80 1 -- 172.21.15.23:0/2649616334 wait complete. 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.349+0000 7fc56fe3db80 1 -- start start 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.349+0000 7fc56fe3db80 1 Processor -- start 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a89c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a89c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a89f850 con 0x55fe3a88fdb0 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a89f9c0 con 0x55fe3a6eb6b0 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe3a89fb80 con 0x55fe3a890b10 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc56fe3db80 10 client.0 _ll_register_callbacks cb 0x55fe3a7a0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc559b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a89c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:43298/0 (socket says 172.21.15.23:43298) 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 -- 172.21.15.23:0/3169322730 learned_addr learned my addr 172.21.15.23:0/3169322730 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55a39b700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a89c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 -- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a6eb6b0 msgr2=0x55fe3a89c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a89c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 -- 172.21.15.23:0/3169322730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 msgr2=0x55fe3a89c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a89c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc55ab9c700 1 -- 172.21.15.23:0/3169322730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe3a7a7130 con 0x55fe3a890b10 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.350+0000 7fc559b9a700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a89c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc55ab9c700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0079f0 tx=0x7fc54c007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78ddf8388659f92 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc53dffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:10.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c00e040 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc544003e70 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc54c003c00 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc54c003dc0 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc56fe3db80 10 client.25456 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fe3a5f7c00 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.351+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc544003e70 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc54c00b0d0 con 0x55fe3a890b10 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 10 client.25456 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 10 client.25456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 20 client.25456 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8f9680 0x55fe3a8fba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 4 client.25456 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:02:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fe3a7af600 con 0x55fe3a8f9680 2023-11-17T00:02:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.352+0000 7fc55a39b700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8f9680 0x55fe3a8fba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.353+0000 7fc55a39b700 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8f9680 0x55fe3a8fba60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67ae52802c9d0b98 server_cookie=61066eef56a209da in_seq=0 out_seq=0 2023-11-17T00:02:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:10.353+0000 7fc53dffb700 10 client.25456 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:10.662 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:10.663 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:10.784 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:10.784 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:10.794 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:10.794 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:10.794 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 15 -lt 100 ']' 2023-11-17T00:02:10.794 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 15 + 1 2023-11-17T00:02:10.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=16 2023-11-17T00:02:10.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:10.909 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:10.910 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:10.912 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:10.913 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:10.913 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc53dffb700 1 -- 172.21.15.23:0/3169322730 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fe3a7af600 con 0x55fe3a8f9680 2023-11-17T00:02:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc53dffb700 10 client.25456 handle_command_reply: tid=0 2023-11-17T00:02:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 client.25456 shutdown 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 msgr2=0x55fe3a89f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0079f0 tx=0x7fc54c007870 comp rx=0 tx=0).stop 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 shutdown_connections 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fe3a890b10 0x55fe3a89f310 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe3a6eb6b0 0x55fe3a89c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55fe3a8f9680 0x55fe3a8fba60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 --2- 172.21.15.23:0/3169322730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe3a88fdb0 0x55fe3a89c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 >> 172.21.15.23:0/3169322730 conn(0x55fe3a884f50 msgr2=0x55fe3a8939c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc53dffb700 0 client.25456 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 shutdown_connections 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 1 -- 172.21.15.23:0/3169322730 wait complete. 2023-11-17T00:02:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.086+0000 7fc56fe3db80 20 client.25456 trim_cache size 0 max 16384 2023-11-17T00:02:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:02:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:02:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:02:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:02:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 msgr2=0x7fc5400a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 0x7fc5400a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc544011490 tx=0x7fc544009260 comp rx=0 tx=0).stop 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 msgr2=0x7fc5541982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e970 tx=0x7fc55000ec80 comp rx=0 tx=0).stop 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 shutdown_connections 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc5400a1150 0x7fc5400a3600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc554104000 0x7fc55419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc554102480 0x7fc5541982f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 --2- 172.21.15.23:0/895084859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc554103670 0x7fc554198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 >> 172.21.15.23:0/895084859 conn(0x7fc5540fdac0 msgr2=0x7fc554106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 shutdown_connections 2023-11-17T00:02:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:11.087+0000 7fc55c5ff700 1 -- 172.21.15.23:0/895084859 wait complete. 2023-11-17T00:02:11.322 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:02:11.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:02:11.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:02:11.587 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:11.587 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:11.665 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:11.666 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:11.673 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:11.673 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:11.673 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 16 -lt 100 ']' 2023-11-17T00:02:11.673 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 16 + 1 2023-11-17T00:02:11.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=17 2023-11-17T00:02:11.707 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:11.714 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:11.714 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:11.718 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:11.719 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:11.720 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:12.288 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:12.288 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:12 smithi173 ceph-mon[131534]: pgmap v402: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:12 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:12 smithi156 ceph-mon[131105]: pgmap v402: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-11-17T00:02:12.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:12 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.417 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:12.417 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:12.453 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:12.453 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:12.453 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 17 -lt 100 ']' 2023-11-17T00:02:12.454 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 17 + 1 2023-11-17T00:02:12.486 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=18 2023-11-17T00:02:12.486 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:12 smithi023 ceph-mon[97824]: pgmap v402: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-11-17T00:02:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:12 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:12.551 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:12.551 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:12.553 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:12.553 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:12.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:13.129 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:13.130 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:13.267 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:13.267 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:13.303 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:13.303 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:13.303 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 18 -lt 100 ']' 2023-11-17T00:02:13.303 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 18 + 1 2023-11-17T00:02:13.336 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=19 2023-11-17T00:02:13.336 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:13.370 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:13.370 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:13.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:13.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:13.374 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:14.412 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:14.412 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:14 smithi023 ceph-mon[97824]: pgmap v403: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-11-17T00:02:14.575 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:14.575 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:14.586 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:14.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:14.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 19 -lt 100 ']' 2023-11-17T00:02:14.587 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 19 + 1 2023-11-17T00:02:14.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=20 2023-11-17T00:02:14.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:14 smithi173 ceph-mon[131534]: pgmap v403: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-11-17T00:02:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:14 smithi156 ceph-mon[131105]: pgmap v403: 129 pgs: 129 active+clean; 755 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-11-17T00:02:14.614 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:14.615 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:14.617 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:14.618 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:14.618 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:15.435 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:15.435 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:15.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:15.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:15.629 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:15.629 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:15.629 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 20 -lt 100 ']' 2023-11-17T00:02:15.629 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 20 + 1 2023-11-17T00:02:15.669 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=21 2023-11-17T00:02:15.669 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:15.698 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:15.699 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:15.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:15.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:15.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:16.237 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:16.237 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:16 smithi023 ceph-mon[97824]: pgmap v404: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 13 op/s 2023-11-17T00:02:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:16 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:16 smithi173 ceph-mon[131534]: pgmap v404: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 13 op/s 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:16 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:16 smithi156 ceph-mon[131105]: pgmap v404: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 13 op/s 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:16 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:16.696 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:16.697 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:16.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:16.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:16.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 21 -lt 100 ']' 2023-11-17T00:02:16.711 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 21 + 1 2023-11-17T00:02:16.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=22 2023-11-17T00:02:16.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:16.772 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:16.773 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:16.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:16.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:16.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:17.419 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:17.420 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:17.558 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:17.558 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:17.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:17.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:17.573 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 22 -lt 100 ']' 2023-11-17T00:02:17.573 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 22 + 1 2023-11-17T00:02:17.602 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=23 2023-11-17T00:02:17.602 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:17.642 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:17.642 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:17.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:17.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:17.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:18.203 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:18.203 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:18.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:18 smithi023 ceph-mon[97824]: pgmap v405: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:18.408 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:18.408 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:18.427 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:18.427 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:18.428 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 23 -lt 100 ']' 2023-11-17T00:02:18.428 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 23 + 1 2023-11-17T00:02:18.460 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=24 2023-11-17T00:02:18.460 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:18.567 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:18.567 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:18.569 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:18.570 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:18.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:18 smithi173 ceph-mon[131534]: pgmap v405: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:18 smithi156 ceph-mon[131105]: pgmap v405: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:19.411 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:19.411 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:19.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:19.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:19.578 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:19.579 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:19.579 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 24 -lt 100 ']' 2023-11-17T00:02:19.579 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 24 + 1 2023-11-17T00:02:19.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=25 2023-11-17T00:02:19.588 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:19.605 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:19.606 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:19.609 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:19.609 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:19.609 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:20.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:19 smithi023 ceph-mon[97824]: pgmap v406: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:19 smithi173 ceph-mon[131534]: pgmap v406: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:19 smithi156 ceph-mon[131105]: pgmap v406: 129 pgs: 129 active+clean; 833 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-11-17T00:02:20.361 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:20.361 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:20.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:20.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:20.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:20.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:20.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 25 -lt 100 ']' 2023-11-17T00:02:20.495 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 25 + 1 2023-11-17T00:02:20.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=26 2023-11-17T00:02:20.544 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:20.584 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:20.585 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:20.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:20.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:20.588 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:21.080 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:21.080 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:21.202 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:21.202 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:21.209 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:21.209 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:21.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 26 -lt 100 ']' 2023-11-17T00:02:21.210 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 26 + 1 2023-11-17T00:02:21.245 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=27 2023-11-17T00:02:21.246 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:21.254 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:21.255 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:21.257 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:21.258 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:21.258 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:21.844 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:21.844 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:22.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:22.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:22.040 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:22.041 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:22.041 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 27 -lt 100 ']' 2023-11-17T00:02:22.041 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 27 + 1 2023-11-17T00:02:22.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=28 2023-11-17T00:02:22.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:22.107 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:22.108 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:22.110 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:22.111 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:22.111 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:22.429 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: pgmap v407: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:22 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:02:22.510 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:22.511 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: pgmap v407: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:22 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: pgmap v407: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:02:22.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:22 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:02:22.657 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:22.658 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:22.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:22.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:22.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 28 -lt 100 ']' 2023-11-17T00:02:22.678 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 28 + 1 2023-11-17T00:02:22.737 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=29 2023-11-17T00:02:22.737 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:22.767 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:22.767 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:22.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:22.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:22.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:23.179 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:23.179 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:23.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:23.300 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:23.310 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:23.311 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:23.311 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 29 -lt 100 ']' 2023-11-17T00:02:23.312 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 29 + 1 2023-11-17T00:02:23.345 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=30 2023-11-17T00:02:23.345 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:23.373 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:23.374 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:23.376 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:23.377 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:23.377 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:23.760 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:23.761 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:23.874 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:23.874 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:23.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:23.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:23.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 30 -lt 100 ']' 2023-11-17T00:02:23.885 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 30 + 1 2023-11-17T00:02:23.917 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=31 2023-11-17T00:02:23.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:23.943 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:23.943 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:23.945 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:23.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:23.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:24.385 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:24.386 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:24 smithi023 ceph-mon[97824]: pgmap v408: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2023-11-17T00:02:24.521 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:24.521 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:24.535 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:24.535 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:24.535 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 31 -lt 100 ']' 2023-11-17T00:02:24.535 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 31 + 1 2023-11-17T00:02:24.576 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=32 2023-11-17T00:02:24.621 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:24.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:24 smithi156 ceph-mon[131105]: pgmap v408: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2023-11-17T00:02:24.622 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:24 smithi173 ceph-mon[131534]: pgmap v408: 129 pgs: 129 active+clean; 893 KiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2023-11-17T00:02:24.636 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:24.636 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:24.639 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:24.640 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:24.640 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:25.394 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:25.394 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:25.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:25.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:25.643 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:25.643 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:25.643 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 32 -lt 100 ']' 2023-11-17T00:02:25.644 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 32 + 1 2023-11-17T00:02:25.670 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=33 2023-11-17T00:02:25.670 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:25.755 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:25.755 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:25.758 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:25.758 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:25.759 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.422+0000 7f49c83ff700 1 -- 172.21.15.23:0/3416985570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 msgr2=0x7f49c0100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.422+0000 7f49c83ff700 1 --2- 172.21.15.23:0/3416985570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0100650 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009990 tx=0x7f49bc009ca0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 -- 172.21.15.23:0/3416985570 shutdown_connections 2023-11-17T00:02:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 --2- 172.21.15.23:0/3416985570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101dc0 0x7f49c0108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 --2- 172.21.15.23:0/3416985570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0100650 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 --2- 172.21.15.23:0/3416985570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101430 0x7f49c0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 -- 172.21.15.23:0/3416985570 >> 172.21.15.23:0/3416985570 conn(0x7f49c00fb840 msgr2=0x7f49c00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 -- 172.21.15.23:0/3416985570 shutdown_connections 2023-11-17T00:02:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.423+0000 7f49c83ff700 1 -- 172.21.15.23:0/3416985570 wait complete. 2023-11-17T00:02:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 Processor -- start 2023-11-17T00:02:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 -- start start 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 0x7f49c0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 0x7f49c019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c019fdc0 con 0x7f49c0100240 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c019ff00 con 0x7f49c0101430 2023-11-17T00:02:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c83ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c01a0090 con 0x7f49c0101dc0 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.424+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36024/0 (socket says 172.21.15.23:36024) 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c699c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 0x7f49c019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 0x7f49c0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 -- 172.21.15.23:0/2242871906 learned_addr learned my addr 172.21.15.23:0/2242871906 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 -- 172.21.15.23:0/2242871906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 msgr2=0x7f49c019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 0x7f49c019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 -- 172.21.15.23:0/2242871906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 msgr2=0x7f49c0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 0x7f49c0198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 -- 172.21.15.23:0/2242871906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0095f0 con 0x7f49c0100240 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c699c700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 0x7f49c019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c599a700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 0x7f49c0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.425+0000 7f49c619b700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0098e0 tx=0x7f49bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.426+0000 7f49b77fe700 1 -- 172.21.15.23:0/2242871906 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc01d070 con 0x7f49c0100240 2023-11-17T00:02:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.426+0000 7f49b77fe700 1 -- 172.21.15.23:0/2242871906 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc005030 con 0x7f49c0100240 2023-11-17T00:02:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.426+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c010bbf0 con 0x7f49c0100240 2023-11-17T00:02:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.426+0000 7f49b77fe700 1 -- 172.21.15.23:0/2242871906 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc004b50 con 0x7f49c0100240 2023-11-17T00:02:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.426+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c010ba60 con 0x7f49c0100240 2023-11-17T00:02:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.429+0000 7f49b77fe700 1 -- 172.21.15.23:0/2242871906 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f49bc004d30 con 0x7f49c0100240 2023-11-17T00:02:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.430+0000 7f49b77fe700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 0x7f49ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.430+0000 7f49b77fe700 1 -- 172.21.15.23:0/2242871906 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f49bc0cdf40 con 0x7f49c0100240 2023-11-17T00:02:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.430+0000 7f49c599a700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 0x7f49ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.431+0000 7f49c599a700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 0x7f49ac0a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f49b0005950 tx=0x7f49b000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.452 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:26.452 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:26.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:26 smithi023 ceph-mon[97824]: pgmap v409: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 Processor -- start 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 -- start start 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc70f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc70ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc614000 con 0x55d5cc56a6b0 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc666840 con 0x55d5cc70edb0 2023-11-17T00:02:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc65fcd0 con 0x55d5cc70fb10 2023-11-17T00:02:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.595+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc70f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36038/0 (socket says 172.21.15.23:36038) 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 -- 172.21.15.23:0/1251411673 learned_addr learned my addr 172.21.15.23:0/1251411673 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c699c700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc70ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 -- 172.21.15.23:0/1251411673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 msgr2=0x55d5cc70ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc70ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 -- 172.21.15.23:0/1251411673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 msgr2=0x55d5cc70f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc70f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 -- 172.21.15.23:0/1251411673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5cc4e15d0 con 0x55d5cc56a6b0 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c599a700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc70f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c699c700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc70ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:02:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49c619b700 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0a2ec0 tx=0x7f49bc09aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ea5904ef4dbb0c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49b4bfe700 1 -- 172.21.15.23:0/1251411673 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a33f0 con 0x55d5cc56a6b0 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49b4bfe700 1 -- 172.21.15.23:0/1251411673 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc0051a0 con 0x55d5cc56a6b0 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.596+0000 7f49b4bfe700 1 -- 172.21.15.23:0/1251411673 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a2f60 con 0x55d5cc56a6b0 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/1251411673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 msgr2=0x55d5cc70e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0a2ec0 tx=0x7f49bc09aca0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/1251411673 shutdown_connections 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc70ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc70e870 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/1251411673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc70f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/1251411673 >> 172.21.15.23:0/1251411673 conn(0x55d5cc703f50 msgr2=0x55d5cc709100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/1251411673 shutdown_connections 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.597+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/1251411673 wait complete. 2023-11-17T00:02:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.598+0000 7f49dbd1ab80 1 -- start start 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.598+0000 7f49dbd1ab80 1 Processor -- start 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.598+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc717e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc601880 con 0x55d5cc56a6b0 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc601520 con 0x55d5cc70edb0 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc4e38c0 con 0x55d5cc70fb10 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49dbd1ab80 10 client.0 _ll_register_callbacks cb 0x55d5cc61f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36044/0 (socket says 172.21.15.23:36044) 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 -- 172.21.15.23:0/203250177 learned_addr learned my addr 172.21.15.23:0/203250177 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c699c700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c599a700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc717e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 -- 172.21.15.23:0/203250177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 msgr2=0x55d5cc71ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 -- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 msgr2=0x55d5cc717e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc717e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 -- 172.21.15.23:0/203250177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c010ba60 con 0x55d5cc56a6b0 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c599a700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc717e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49c619b700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49bc09b1f0 tx=0x7f49bc097370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae55ad1fbc5de99e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.599+0000 7f49a9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:02:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a2f60 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f49bc0a30c0 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5cc701710 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49bc0b9250 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 10 client.15414 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d5cc626130 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc0cb020 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49bc0cbc70 con 0x55d5cc56a6b0 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 10 client.15414 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 10 client.15414 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 20 client.15414 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc7231a0 0x55d5cc7255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 4 client.15414 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.600+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5cc622040 con 0x55d5cc7231a0 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.601+0000 7f49c599a700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc7231a0 0x55d5cc7255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49c599a700 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc7231a0 0x55d5cc7255b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e936b41067f83fa0 server_cookie=f8f0da1b353eeff0 in_seq=0 out_seq=0 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49a9ffb700 10 client.15414 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49a9ffb700 1 -- 172.21.15.23:0/203250177 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d5cc622040 con 0x55d5cc7231a0 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49a9ffb700 10 client.15414 handle_command_reply: tid=0 2023-11-17T00:02:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49dbd1ab80 1 client.15414 shutdown 2023-11-17T00:02:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.602+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 msgr2=0x55d5cc7178c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49bc09b1f0 tx=0x7f49bc097370 comp rx=0 tx=0).stop 2023-11-17T00:02:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 shutdown_connections 2023-11-17T00:02:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc7231a0 0x55d5cc7255b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc56a6b0 0x55d5cc7178c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/203250177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc717e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49a9ffb700 0 client.15414 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 >> 172.21.15.23:0/203250177 conn(0x55d5cc703f50 msgr2=0x55d5cc7087e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 shutdown_connections 2023-11-17T00:02:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/203250177 wait complete. 2023-11-17T00:02:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.603+0000 7f49dbd1ab80 20 client.15414 trim_cache size 0 max 16384 2023-11-17T00:02:26.612 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:26 smithi156 ceph-mon[131105]: pgmap v409: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2023-11-17T00:02:26.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:26 smithi173 ceph-mon[131534]: pgmap v409: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2023-11-17T00:02:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 Processor -- start 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 -- start start 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70edb0 0x55d5cc729f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc705420 0x55d5cc717670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc70ac10 con 0x55d5cc705420 2023-11-17T00:02:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc70f450 con 0x55d5cc70fb10 2023-11-17T00:02:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc709aa0 con 0x55d5cc70edb0 2023-11-17T00:02:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49c599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49c599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34960/0 (socket says 172.21.15.23:34960) 2023-11-17T00:02:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49c599a700 1 -- 172.21.15.23:0/4224254727 learned_addr learned my addr 172.21.15.23:0/4224254727 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49c699c700 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc705420 0x55d5cc717670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.619+0000 7f49c619b700 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70edb0 0x55d5cc729f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 -- 172.21.15.23:0/4224254727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70edb0 msgr2=0x55d5cc729f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70edb0 0x55d5cc729f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 -- 172.21.15.23:0/4224254727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc705420 msgr2=0x55d5cc717670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc705420 0x55d5cc717670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 -- 172.21.15.23:0/4224254727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5cc625e80 con 0x55d5cc70fb10 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49c599a700 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f49b000be70 tx=0x7f49b0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9b82df64eff5e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49b53ff700 1 -- 172.21.15.23:0/4224254727 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b002b070 con 0x55d5cc70fb10 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49b53ff700 1 -- 172.21.15.23:0/4224254727 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49b0015070 con 0x55d5cc70fb10 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.620+0000 7f49b53ff700 1 -- 172.21.15.23:0/4224254727 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0025040 con 0x55d5cc70fb10 2023-11-17T00:02:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4224254727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 msgr2=0x55d5cc71ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f49b000be70 tx=0x7f49b0016000 comp rx=0 tx=0).stop 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4224254727 shutdown_connections 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc705420 0x55d5cc717670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc70edb0 0x55d5cc729f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4224254727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70fb10 0x55d5cc71ee00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4224254727 >> 172.21.15.23:0/4224254727 conn(0x55d5cc703f50 msgr2=0x55d5cc7090d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4224254727 shutdown_connections 2023-11-17T00:02:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.621+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4224254727 wait complete. 2023-11-17T00:02:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 -- start start 2023-11-17T00:02:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 Processor -- start 2023-11-17T00:02:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc71eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc70fb10 0x55d5cc71b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc71b580 con 0x55d5cc70fb10 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc71b6f0 con 0x55d5cc70edb0 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5cc71b8b0 con 0x55d5cc705420 2023-11-17T00:02:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.622+0000 7f49dbd1ab80 10 client.0 _ll_register_callbacks cb 0x7f49b000bd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c699c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc70fb10 0x55d5cc71b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:34114/0 (socket says 172.21.15.23:34114) 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 -- 172.21.15.23:0/4164060931 learned_addr learned my addr 172.21.15.23:0/4164060931 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c599a700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc71eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 -- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 msgr2=0x55d5cc71eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc71eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 -- 172.21.15.23:0/4164060931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc70fb10 msgr2=0x55d5cc71b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc70fb10 0x55d5cc71b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 -- 172.21.15.23:0/4164060931 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b0011440 con 0x55d5cc705420 2023-11-17T00:02:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49c619b700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f49bc099380 tx=0x7f49bc09b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511473e7e73a92d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49a9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.623+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a3160 con 0x55d5cc705420 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49bc0051a0 con 0x55d5cc705420 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f49bc09b030 con 0x55d5cc705420 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49bc09ab70 con 0x55d5cc705420 2023-11-17T00:02:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 10 client.25474 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d5cc476c00 con 0x55d5cc705420 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc0051a0 con 0x55d5cc705420 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49bc0cbb50 con 0x55d5cc705420 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 10 client.25474 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 10 client.25474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 20 client.25474 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc777ec0 0x55d5cc77a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 4 client.25474 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.624+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d5cc62e600 con 0x55d5cc777ec0 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.625+0000 7f49c599a700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc777ec0 0x55d5cc77a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.625+0000 7f49c599a700 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc777ec0 0x55d5cc77a2a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7b1d9cdbbe07b3a server_cookie=1f2e00bc0dad0da0 in_seq=0 out_seq=0 2023-11-17T00:02:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.625+0000 7f49a9ffb700 10 client.25474 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49a9ffb700 1 -- 172.21.15.23:0/4164060931 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d5cc62e600 con 0x55d5cc777ec0 2023-11-17T00:02:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49a9ffb700 10 client.25474 handle_command_reply: tid=0 2023-11-17T00:02:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49dbd1ab80 1 client.25474 shutdown 2023-11-17T00:02:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 msgr2=0x55d5cc71e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f49bc099380 tx=0x7f49bc09b4a0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 shutdown_connections 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.642+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d5cc777ec0 0x55d5cc77a2a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d5cc70fb10 0x55d5cc71b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49a9ffb700 0 client.25474 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5cc705420 0x55d5cc71e5a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 --2- 172.21.15.23:0/4164060931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5cc70edb0 0x55d5cc71eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 >> 172.21.15.23:0/4164060931 conn(0x55d5cc703f50 msgr2=0x55d5cc717670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 shutdown_connections 2023-11-17T00:02:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 1 -- 172.21.15.23:0/4164060931 wait complete. 2023-11-17T00:02:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49dbd1ab80 20 client.25474 trim_cache size 0 max 16384 2023-11-17T00:02:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:02:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:02:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:02:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:02:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 msgr2=0x7f49ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 0x7f49ac0a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f49b0005950 tx=0x7f49b000b410 comp rx=0 tx=0).stop 2023-11-17T00:02:26.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 msgr2=0x7f49c0198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.643+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0098e0 tx=0x7f49bc004730 comp rx=0 tx=0).stop 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 shutdown_connections 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f49ac0a1150 0x7f49ac0a3600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0101dc0 0x7f49c019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49c0100240 0x7f49c0198240 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 --2- 172.21.15.23:0/2242871906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49c0101430 0x7f49c0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 >> 172.21.15.23:0/2242871906 conn(0x7f49c00fb840 msgr2=0x7f49c0106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 shutdown_connections 2023-11-17T00:02:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:26.644+0000 7f49c83ff700 1 -- 172.21.15.23:0/2242871906 wait complete. 2023-11-17T00:02:26.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:26.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:26.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:26.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:26.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 33 -lt 100 ']' 2023-11-17T00:02:26.692 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 33 + 1 2023-11-17T00:02:26.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=34 2023-11-17T00:02:26.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:26.890 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:26.890 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:26.892 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:26.893 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:26.893 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:27.476 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:27.477 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:27.604 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:27.604 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:27.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:27.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:27.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 34 -lt 100 ']' 2023-11-17T00:02:27.617 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 34 + 1 2023-11-17T00:02:27.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=35 2023-11-17T00:02:27.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:27.680 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:27.681 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:27.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:27.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:27.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:28.118 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:28.118 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:28.240 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:28.240 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:28.250 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:28.250 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:28.251 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 35 -lt 100 ']' 2023-11-17T00:02:28.251 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 35 + 1 2023-11-17T00:02:28.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=36 2023-11-17T00:02:28.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:28.315 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:28.315 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:28.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:28.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:28.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:28 smithi023 ceph-mon[97824]: pgmap v410: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-11-17T00:02:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:28 smithi173 ceph-mon[131534]: pgmap v410: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-11-17T00:02:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:28 smithi156 ceph-mon[131105]: pgmap v410: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-11-17T00:02:28.743 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:28.743 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:28.856 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:28.857 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:28.876 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:28.876 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:28.876 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 36 -lt 100 ']' 2023-11-17T00:02:28.876 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 36 + 1 2023-11-17T00:02:28.911 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=37 2023-11-17T00:02:28.911 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:28.934 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:28.934 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:28.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:28.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:28.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:29.435 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:29.435 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:29.631 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:29.631 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:29.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:29.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:29.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 37 -lt 100 ']' 2023-11-17T00:02:29.653 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 37 + 1 2023-11-17T00:02:29.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=38 2023-11-17T00:02:29.705 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:29.760 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:29.761 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:29.765 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:29.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:29.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:29 smithi173 ceph-mon[131534]: pgmap v411: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2023-11-17T00:02:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:29 smithi156 ceph-mon[131105]: pgmap v411: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2023-11-17T00:02:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:29 smithi023 ceph-mon[97824]: pgmap v411: 129 pgs: 129 active+clean; 952 KiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 13 op/s 2023-11-17T00:02:31.709 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:31.710 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:31.922 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:31.923 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:31.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:31.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:31.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 38 -lt 100 ']' 2023-11-17T00:02:31.952 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 38 + 1 2023-11-17T00:02:31.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=39 2023-11-17T00:02:31.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:32.018 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:32.018 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:32.020 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:32.021 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:32.022 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:32.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:32 smithi023 ceph-mon[97824]: pgmap v412: 129 pgs: 129 active+clean; 1.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-11-17T00:02:32.559 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:32.560 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:32 smithi173 ceph-mon[131534]: pgmap v412: 129 pgs: 129 active+clean; 1.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-11-17T00:02:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:32 smithi156 ceph-mon[131105]: pgmap v412: 129 pgs: 129 active+clean; 1.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-11-17T00:02:32.773 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:32.773 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:32.793 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:32.793 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:32.793 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 39 -lt 100 ']' 2023-11-17T00:02:32.793 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 39 + 1 2023-11-17T00:02:32.825 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=40 2023-11-17T00:02:32.826 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:32.860 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:32.861 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:32.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:32.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:32.864 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:33.409 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:33.410 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:34.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:34.093 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.094 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.094 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 40 -lt 100 ']' 2023-11-17T00:02:34.094 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 40 + 1 2023-11-17T00:02:34.125 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=41 2023-11-17T00:02:34.125 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:34.160 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:34.160 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:34.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:34.164 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:34.448 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:34 smithi023 ceph-mon[97824]: pgmap v413: 129 pgs: 129 active+clean; 1.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 15 op/s 2023-11-17T00:02:34.592 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:34.593 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:34 smithi173 ceph-mon[131534]: pgmap v413: 129 pgs: 129 active+clean; 1.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 15 op/s 2023-11-17T00:02:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:34 smithi156 ceph-mon[131105]: pgmap v413: 129 pgs: 129 active+clean; 1.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 15 op/s 2023-11-17T00:02:34.731 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:34.731 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:34.749 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.749 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.749 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 41 -lt 100 ']' 2023-11-17T00:02:34.750 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 41 + 1 2023-11-17T00:02:34.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=42 2023-11-17T00:02:34.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:34.813 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:34.814 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:34.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:34.826 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:34.939 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:34.939 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.963 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:34.964 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:34.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:34.970 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 42 -lt 100 ']' 2023-11-17T00:02:34.970 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 42 + 1 2023-11-17T00:02:34.977 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=43 2023-11-17T00:02:34.977 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:34.987 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:34.988 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:34.990 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:34.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:34.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:35.107 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:35.109 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:35.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:35.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:35.147 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:35.147 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:35.147 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 43 -lt 100 ']' 2023-11-17T00:02:35.147 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 43 + 1 2023-11-17T00:02:35.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=44 2023-11-17T00:02:35.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:35.173 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:35.173 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:35.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:35.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:35.179 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:35.350 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:35.351 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:35.491 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:35.491 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:35.499 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:35.499 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:35.499 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 44 -lt 100 ']' 2023-11-17T00:02:35.499 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 44 + 1 2023-11-17T00:02:35.523 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=45 2023-11-17T00:02:35.523 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:35.540 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:35.540 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:35.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:35.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:35.544 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:35.867 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:35.867 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:36.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:36.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:36.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:36.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:36.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 45 -lt 100 ']' 2023-11-17T00:02:36.016 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 45 + 1 2023-11-17T00:02:36.057 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=46 2023-11-17T00:02:36.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:36.090 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:36.090 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:36.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:36.093 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:36.093 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:36.475 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:36.475 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:36 smithi023 ceph-mon[97824]: pgmap v414: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2023-11-17T00:02:36.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:36.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:36.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:36 smithi173 ceph-mon[131534]: pgmap v414: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2023-11-17T00:02:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:36 smithi156 ceph-mon[131105]: pgmap v414: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2023-11-17T00:02:36.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:36.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:36.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 46 -lt 100 ']' 2023-11-17T00:02:36.616 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 46 + 1 2023-11-17T00:02:36.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=47 2023-11-17T00:02:36.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:36.683 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:36.683 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:36.685 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:36.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:36.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:37.059 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:37.059 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:37.190 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:37.191 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:37.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:37.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:37.199 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 47 -lt 100 ']' 2023-11-17T00:02:37.199 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 47 + 1 2023-11-17T00:02:37.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=48 2023-11-17T00:02:37.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:37.272 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:37.272 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:37.274 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:37.275 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:37.276 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:37.741 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:37.742 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:37.888 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:37.888 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:37.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:37.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:37.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 48 -lt 100 ']' 2023-11-17T00:02:37.901 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 48 + 1 2023-11-17T00:02:37.932 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=49 2023-11-17T00:02:37.932 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:37.963 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:37.964 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:37.966 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:37.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:37.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:38.325 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:38.325 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:38.446 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:38.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:38.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:38.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:38.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 49 -lt 100 ']' 2023-11-17T00:02:38.458 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 49 + 1 2023-11-17T00:02:38.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=50 2023-11-17T00:02:38.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:38 smithi023 ceph-mon[97824]: pgmap v415: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 15 op/s 2023-11-17T00:02:38.519 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:38.519 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:38.521 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:38.522 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:38.522 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:38 smithi173 ceph-mon[131534]: pgmap v415: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 15 op/s 2023-11-17T00:02:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:38 smithi156 ceph-mon[131105]: pgmap v415: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 15 op/s 2023-11-17T00:02:38.867 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:38.867 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:39.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:39.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:39.021 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:39.023 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:39.023 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 50 -lt 100 ']' 2023-11-17T00:02:39.023 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 50 + 1 2023-11-17T00:02:39.051 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=51 2023-11-17T00:02:39.052 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:39.080 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:39.080 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:39.082 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:39.083 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:39.083 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:39.508 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:39.509 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:39.697 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:39.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:39.715 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:39.716 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:39.716 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 51 -lt 100 ']' 2023-11-17T00:02:39.716 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 51 + 1 2023-11-17T00:02:39.749 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=52 2023-11-17T00:02:39.749 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:39.813 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:39.814 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:39.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:39.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:39.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:40.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:39 smithi173 ceph-mon[131534]: pgmap v416: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2023-11-17T00:02:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:39 smithi156 ceph-mon[131105]: pgmap v416: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2023-11-17T00:02:40.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:39 smithi023 ceph-mon[97824]: pgmap v416: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 13 op/s 2023-11-17T00:02:40.434 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:40.434 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:40.638 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:40.638 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:40.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:40.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:40.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 52 -lt 100 ']' 2023-11-17T00:02:40.685 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 52 + 1 2023-11-17T00:02:40.715 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=53 2023-11-17T00:02:40.715 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:40.810 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:40.811 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:40.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:40.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:40.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:41.424 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:41.425 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:41.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:41.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:41.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:41.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:41.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 53 -lt 100 ']' 2023-11-17T00:02:41.703 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 53 + 1 2023-11-17T00:02:41.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=54 2023-11-17T00:02:41.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:41.865 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:41.866 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:41.869 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:41.870 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:41.871 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.036+0000 7eff87fff700 1 -- 172.21.15.23:0/243228753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 msgr2=0x7eff80101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.036+0000 7eff87fff700 1 --2- 172.21.15.23:0/243228753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff80101900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7eff70009990 tx=0x7eff70009ca0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 -- 172.21.15.23:0/243228753 shutdown_connections 2023-11-17T00:02:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 --2- 172.21.15.23:0/243228753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff80108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 --2- 172.21.15.23:0/243228753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 --2- 172.21.15.23:0/243228753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff80101900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 -- 172.21.15.23:0/243228753 >> 172.21.15.23:0/243228753 conn(0x7eff800fb8e0 msgr2=0x7eff800fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 -- 172.21.15.23:0/243228753 shutdown_connections 2023-11-17T00:02:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.037+0000 7eff87fff700 1 -- 172.21.15.23:0/243228753 wait complete. 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 Processor -- start 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 -- start start 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff801986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff8019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8019fd20 con 0x7eff801002c0 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.038+0000 7eff87fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8019fe60 con 0x7eff80101e40 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff87fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8019fff0 con 0x7eff801014b0 2023-11-17T00:02:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32840/0 (socket says 172.21.15.23:32840) 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 -- 172.21.15.23:0/1285083092 learned_addr learned my addr 172.21.15.23:0/1285083092 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff8559a700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff801986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff8659c700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff8019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 -- 172.21.15.23:0/1285083092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 msgr2=0x7eff801986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff801986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 -- 172.21.15.23:0/1285083092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 msgr2=0x7eff8019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff8019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff85d9b700 1 -- 172.21.15.23:0/1285083092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700095f0 con 0x7eff801002c0 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff8559a700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff801986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.039+0000 7eff8659c700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff8019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00e970 tx=0x7eff7c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff76ffd700 1 -- 172.21.15.23:0/1285083092 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c009790 con 0x7eff801002c0 2023-11-17T00:02:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff76ffd700 1 -- 172.21.15.23:0/1285083092 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff7c00eec0 con 0x7eff801002c0 2023-11-17T00:02:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8010bba0 con 0x7eff801002c0 2023-11-17T00:02:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff76ffd700 1 -- 172.21.15.23:0/1285083092 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c003cf0 con 0x7eff801002c0 2023-11-17T00:02:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.040+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8010ba10 con 0x7eff801002c0 2023-11-17T00:02:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.044+0000 7eff76ffd700 1 -- 172.21.15.23:0/1285083092 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7eff7c00ccd0 con 0x7eff801002c0 2023-11-17T00:02:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.045+0000 7eff76ffd700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 0x7eff6c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.045+0000 7eff76ffd700 1 -- 172.21.15.23:0/1285083092 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7eff7c014070 con 0x7eff801002c0 2023-11-17T00:02:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.045+0000 7eff8559a700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 0x7eff6c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.046+0000 7eff8559a700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 0x7eff6c0a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b2e0 tx=0x7eff70005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.092+0000 7eff9b90cb80 1 Processor -- start 2023-11-17T00:02:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 -- start start 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fdc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 0x55a5a1fddf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1ee2000 con 0x55a5a1fdcdb0 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1f34840 con 0x55a5a1fddb10 2023-11-17T00:02:42.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.093+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1f2dcd0 con 0x55a5a1e386b0 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 0x55a5a1fddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fdc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32846/0 (socket says 172.21.15.23:32846) 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8559a700 1 -- 172.21.15.23:0/507843909 learned_addr learned my addr 172.21.15.23:0/507843909 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 -- 172.21.15.23:0/507843909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 msgr2=0x55a5a1fdc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fdc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 -- 172.21.15.23:0/507843909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 msgr2=0x55a5a1fdd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8659c700 1 -- 172.21.15.23:0/507843909 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5a1daf5d0 con 0x55a5a1fddb10 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.094+0000 7eff8559a700 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:42.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.095+0000 7eff8659c700 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 0x55a5a1fddf80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7eff780093a0 tx=0x7eff78009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63cf0fb34ee4d15d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.095+0000 7eff6bfff700 1 -- 172.21.15.23:0/507843909 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800e070 con 0x55a5a1fddb10 2023-11-17T00:02:42.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.095+0000 7eff6bfff700 1 -- 172.21.15.23:0/507843909 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff7800f040 con 0x55a5a1fddb10 2023-11-17T00:02:42.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.095+0000 7eff6bfff700 1 -- 172.21.15.23:0/507843909 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff78015e60 con 0x55a5a1fddb10 2023-11-17T00:02:42.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 -- 172.21.15.23:0/507843909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 msgr2=0x55a5a1fddf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 0x55a5a1fddf80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7eff780093a0 tx=0x7eff78009920 comp rx=0 tx=0).stop 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 -- 172.21.15.23:0/507843909 shutdown_connections 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fddb10 0x55a5a1fddf80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fdc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/507843909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fdd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 -- 172.21.15.23:0/507843909 >> 172.21.15.23:0/507843909 conn(0x55a5a1fd1f50 msgr2=0x55a5a1fd7100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 -- 172.21.15.23:0/507843909 shutdown_connections 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.096+0000 7eff9b90cb80 1 -- 172.21.15.23:0/507843909 wait complete. 2023-11-17T00:02:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.097+0000 7eff9b90cb80 1 -- start start 2023-11-17T00:02:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.097+0000 7eff9b90cb80 1 Processor -- start 2023-11-17T00:02:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1ecf880 con 0x55a5a1fddb10 2023-11-17T00:02:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1ecf520 con 0x55a5a1fdcdb0 2023-11-17T00:02:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1db18c0 con 0x55a5a1e386b0 2023-11-17T00:02:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff9b90cb80 10 client.0 _ll_register_callbacks cb 0x55a5a1eed720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff8559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:49394/0 (socket says 172.21.15.23:49394) 2023-11-17T00:02:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.098+0000 7eff85d9b700 1 -- 172.21.15.23:0/2090617195 learned_addr learned my addr 172.21.15.23:0/2090617195 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff8659c700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 -- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fdcdb0 msgr2=0x55a5a1fe6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 -- 172.21.15.23:0/2090617195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 msgr2=0x55a5a1fed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fed1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 -- 172.21.15.23:0/2090617195 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff78009050 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff8659c700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff85d9b700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c5b0 tx=0x7eff7c0163b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a43dc6431673a46 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff697fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c099660 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7eff7c0a23c0 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.099+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5a1fcf710 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff7c0a1f30 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 10 client.25486 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a5a1ef4130 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5a1fcf710 con 0x55a5a1e386b0 2023-11-17T00:02:42.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7eff7c095e80 con 0x55a5a1e386b0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 10 client.25486 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 10 client.25486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 20 client.25486 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a1ff19a0 0x55a5a1ff3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 4 client.25486 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.100+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5a1ef0040 con 0x55a5a1ff19a0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.101+0000 7eff8559a700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a1ff19a0 0x55a5a1ff3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.102+0000 7eff8559a700 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a1ff19a0 0x55a5a1ff3db0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de73b23b81a96c5e server_cookie=e0ad07a77cf263de in_seq=0 out_seq=0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.102+0000 7eff697fa700 10 client.25486 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.102+0000 7eff697fa700 1 -- 172.21.15.23:0/2090617195 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a5a1ef0040 con 0x55a5a1ff19a0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.102+0000 7eff697fa700 10 client.25486 handle_command_reply: tid=0 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.102+0000 7eff9b90cb80 1 client.25486 shutdown 2023-11-17T00:02:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 msgr2=0x55a5a1fe5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c5b0 tx=0x7eff7c0163b0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 shutdown_connections 2023-11-17T00:02:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a1ff19a0 0x55a5a1ff3db0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe5c20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff697fa700 0 client.25486 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/2090617195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 >> 172.21.15.23:0/2090617195 conn(0x55a5a1fd1f50 msgr2=0x55a5a1fd67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 shutdown_connections 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 1 -- 172.21.15.23:0/2090617195 wait complete. 2023-11-17T00:02:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.103+0000 7eff9b90cb80 20 client.25486 trim_cache size 0 max 16384 2023-11-17T00:02:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 Processor -- start 2023-11-17T00:02:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 -- start start 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fed160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fe5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1fe57f0 con 0x55a5a1fddb10 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1ff17c0 con 0x55a5a1e386b0 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.121+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1fcf280 con 0x55a5a1fdcdb0 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff8659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fe5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:59990/0 (socket says 172.21.15.23:59990) 2023-11-17T00:02:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 -- 172.21.15.23:0/1413899775 learned_addr learned my addr 172.21.15.23:0/1413899775 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff8559a700 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 -- 172.21.15.23:0/1413899775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fdcdb0 msgr2=0x55a5a1fed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 -- 172.21.15.23:0/1413899775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 msgr2=0x55a5a1fe5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fe5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 -- 172.21.15.23:0/1413899775 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5a1ef3e80 con 0x55a5a1e386b0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff8659c700 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.122+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0969e0 tx=0x7eff7c098520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e967938d7875d0f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff74ff9700 1 -- 172.21.15.23:0/1413899775 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c098f70 con 0x55a5a1e386b0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff74ff9700 1 -- 172.21.15.23:0/1413899775 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff7c0a23c0 con 0x55a5a1e386b0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff74ff9700 1 -- 172.21.15.23:0/1413899775 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c095fc0 con 0x55a5a1e386b0 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1413899775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 msgr2=0x55a5a1fe59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0969e0 tx=0x7eff7c098520 comp rx=0 tx=0).stop 2023-11-17T00:02:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1413899775 shutdown_connections 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fddb10 0x55a5a1fe5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe59d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.123+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1413899775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.124+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1413899775 >> 172.21.15.23:0/1413899775 conn(0x55a5a1fd1f50 msgr2=0x55a5a1fe0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.124+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1413899775 shutdown_connections 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.124+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1413899775 wait complete. 2023-11-17T00:02:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.124+0000 7eff9b90cb80 1 -- start start 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.124+0000 7eff9b90cb80 1 Processor -- start 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 0x55a5a1fec650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1fecb90 con 0x55a5a1fdcdb0 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1fecd00 con 0x55a5a1e386b0 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a1fecec0 con 0x55a5a1fddb10 2023-11-17T00:02:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff9b90cb80 10 client.0 _ll_register_callbacks cb 0x7eff7c099ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff8559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff85d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff8659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 0x55a5a1fec650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff8559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32872/0 (socket says 172.21.15.23:32872) 2023-11-17T00:02:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.125+0000 7eff8559a700 1 -- 172.21.15.23:0/1613023783 learned_addr learned my addr 172.21.15.23:0/1613023783 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 -- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 msgr2=0x55a5a1fe90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 -- 172.21.15.23:0/1613023783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 msgr2=0x55a5a1fe9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 -- 172.21.15.23:0/1613023783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5a1daf5d0 con 0x55a5a1fddb10 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8559a700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff8659c700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 0x55a5a1fec650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff78009370 tx=0x7eff78015670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4928ad7e768b33b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff697fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.126+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800e040 con 0x55a5a1fddb10 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7eff78015a70 con 0x55a5a1fddb10 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff7800f040 con 0x55a5a1fddb10 2023-11-17T00:02:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff78015c30 con 0x55a5a1fddb10 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff9b90cb80 10 client.25492 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a5a1ef4130 con 0x55a5a1fddb10 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.127+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff7800f040 con 0x55a5a1fddb10 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7eff78005040 con 0x55a5a1fddb10 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 10 client.25492 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 10 client.25492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 20 client.25492 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a20469b0 0x55a5a2048dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 4 client.25492 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a5a1efc600 con 0x55a5a20469b0 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.128+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a20469b0 0x55a5a2048dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.129+0000 7eff85d9b700 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a20469b0 0x55a5a2048dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4923a0e9dfa59343 server_cookie=9e713c11be7c499 in_seq=0 out_seq=0 2023-11-17T00:02:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.129+0000 7eff697fa700 10 client.25492 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.148+0000 7eff697fa700 1 -- 172.21.15.23:0/1613023783 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a5a1efc600 con 0x55a5a20469b0 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.148+0000 7eff697fa700 10 client.25492 handle_command_reply: tid=0 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.148+0000 7eff9b90cb80 1 client.25492 shutdown 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 msgr2=0x55a5a1fec650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 0x55a5a1fec650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff78009370 tx=0x7eff78015670 comp rx=0 tx=0).stop 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 shutdown_connections 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a5a1fddb10 0x55a5a1fec650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5a1e386b0 0x55a5a1fe90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55a5a20469b0 0x55a5a2048dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 --2- 172.21.15.23:0/1613023783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a5a1fdcdb0 0x55a5a1fe9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff697fa700 0 client.25492 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 >> 172.21.15.23:0/1613023783 conn(0x55a5a1fd1f50 msgr2=0x55a5a1fe59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 shutdown_connections 2023-11-17T00:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 1 -- 172.21.15.23:0/1613023783 wait complete. 2023-11-17T00:02:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.149+0000 7eff9b90cb80 20 client.25492 trim_cache size 0 max 16384 2023-11-17T00:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:02:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 msgr2=0x7eff6c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 0x7eff6c0a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b2e0 tx=0x7eff70005fb0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 msgr2=0x7eff801981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00e970 tx=0x7eff7c00ec80 comp rx=0 tx=0).stop 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 shutdown_connections 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff80101e40 0x7eff8019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff801002c0 0x7eff801981a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7eff6c0a1150 0x7eff6c0a3600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 --2- 172.21.15.23:0/1285083092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff801014b0 0x7eff801986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 >> 172.21.15.23:0/1285083092 conn(0x7eff800fb8e0 msgr2=0x7eff800fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 shutdown_connections 2023-11-17T00:02:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:42.150+0000 7eff87fff700 1 -- 172.21.15.23:0/1285083092 wait complete. 2023-11-17T00:02:42.466 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:42.466 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:42 smithi023 ceph-mon[97824]: pgmap v417: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-17T00:02:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:42 smithi173 ceph-mon[131534]: pgmap v417: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-17T00:02:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:42 smithi156 ceph-mon[131105]: pgmap v417: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-17T00:02:42.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:42.647 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:42.664 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:42.664 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:42.665 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 54 -lt 100 ']' 2023-11-17T00:02:42.665 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 54 + 1 2023-11-17T00:02:42.698 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=55 2023-11-17T00:02:42.698 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:42.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:42.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:42.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:42.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:42.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:43.108 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:43.108 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:43.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:43.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:43.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:43.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:43.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 55 -lt 100 ']' 2023-11-17T00:02:43.266 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 55 + 1 2023-11-17T00:02:43.298 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=56 2023-11-17T00:02:43.298 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:43.330 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:43.330 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:43.332 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:43.333 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:43.335 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:43.802 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:43.802 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:43.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:43.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:43.940 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:43.940 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:43.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 56 -lt 100 ']' 2023-11-17T00:02:43.941 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 56 + 1 2023-11-17T00:02:43.990 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=57 2023-11-17T00:02:43.990 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:44.062 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:44.063 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:44.066 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:44.066 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:44.067 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:44.424 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:44.424 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:44.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:44 smithi023 ceph-mon[97824]: pgmap v418: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-17T00:02:44.537 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:44.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:44.548 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:44.548 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:44.548 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 57 -lt 100 ']' 2023-11-17T00:02:44.548 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 57 + 1 2023-11-17T00:02:44.581 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=58 2023-11-17T00:02:44.581 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:44.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:44 smithi173 ceph-mon[131534]: pgmap v418: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-17T00:02:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:44 smithi156 ceph-mon[131105]: pgmap v418: 129 pgs: 129 active+clean; 1.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-17T00:02:44.613 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:44.613 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:44.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:44.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:44.617 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:45.008 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:45.009 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:45.112 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:45.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:45.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:45.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:45.125 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 58 -lt 100 ']' 2023-11-17T00:02:45.125 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 58 + 1 2023-11-17T00:02:45.156 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=59 2023-11-17T00:02:45.157 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:45.183 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:45.183 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:45.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:45.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:45.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:45.524 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:45.524 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:45.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:45.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:45.658 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:45.659 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:45.659 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 59 -lt 100 ']' 2023-11-17T00:02:45.659 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 59 + 1 2023-11-17T00:02:45.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=60 2023-11-17T00:02:45.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:45.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:45.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:45.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:45.725 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:45.725 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:46.099 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:46.100 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:46.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:46.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:46.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:46.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:46.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 60 -lt 100 ']' 2023-11-17T00:02:46.249 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 60 + 1 2023-11-17T00:02:46.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=61 2023-11-17T00:02:46.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:46.313 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:46.313 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:46.314 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:46.314 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:46.315 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:46 smithi023 ceph-mon[97824]: pgmap v419: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 14 op/s 2023-11-17T00:02:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:46 smithi173 ceph-mon[131534]: pgmap v419: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 14 op/s 2023-11-17T00:02:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:46 smithi156 ceph-mon[131105]: pgmap v419: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 14 op/s 2023-11-17T00:02:46.700 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:46.700 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:46.804 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:46.804 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:46.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:46.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:46.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 61 -lt 100 ']' 2023-11-17T00:02:46.814 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 61 + 1 2023-11-17T00:02:46.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=62 2023-11-17T00:02:46.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:46.873 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:46.874 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:46.876 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:46.879 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:46.879 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:47.249 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:47.249 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:47.379 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:47.379 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:47.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:47.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:47.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 62 -lt 100 ']' 2023-11-17T00:02:47.391 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 62 + 1 2023-11-17T00:02:47.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=63 2023-11-17T00:02:47.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:47.453 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:47.453 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:47.456 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:47.456 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:47.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:47.924 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:47.924 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:48.051 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:48.051 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:48.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:48.064 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:48.064 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 63 -lt 100 ']' 2023-11-17T00:02:48.064 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 63 + 1 2023-11-17T00:02:48.097 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=64 2023-11-17T00:02:48.097 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:48.127 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:48.127 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:48.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:48.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:48.131 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:48.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:48 smithi023 ceph-mon[97824]: pgmap v420: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:48.582 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:48.582 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:48 smithi173 ceph-mon[131534]: pgmap v420: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:48 smithi156 ceph-mon[131105]: pgmap v420: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:48.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:48.679 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:48.690 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:48.691 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:48.691 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 64 -lt 100 ']' 2023-11-17T00:02:48.691 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 64 + 1 2023-11-17T00:02:48.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=65 2023-11-17T00:02:48.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:48.748 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:48.748 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:48.750 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:48.751 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:48.751 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:49.100 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:49.100 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:49.226 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:49.226 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:49.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:49.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:49.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 65 -lt 100 ']' 2023-11-17T00:02:49.242 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 65 + 1 2023-11-17T00:02:49.272 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=66 2023-11-17T00:02:49.272 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:49.303 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:49.303 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:49.306 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:49.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:49.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:49.691 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:49.691 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:49.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:49.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:49.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:49.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:49.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 66 -lt 100 ']' 2023-11-17T00:02:49.801 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 66 + 1 2023-11-17T00:02:49.830 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=67 2023-11-17T00:02:49.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:49.857 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:49.857 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:49.859 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:49.860 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:49.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:50.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:49 smithi023 ceph-mon[97824]: pgmap v421: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:49 smithi156 ceph-mon[131105]: pgmap v421: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:49 smithi173 ceph-mon[131534]: pgmap v421: 129 pgs: 129 active+clean; 1.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 14 op/s 2023-11-17T00:02:50.218 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:50.218 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:50.385 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:50.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:50.402 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:50.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:50.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 67 -lt 100 ']' 2023-11-17T00:02:50.403 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 67 + 1 2023-11-17T00:02:50.430 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=68 2023-11-17T00:02:50.430 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:50.461 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:50.462 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:50.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:50.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:50.465 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:50.848 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:50.849 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:50.976 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:50.976 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:50.989 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:50.989 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:50.989 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 68 -lt 100 ']' 2023-11-17T00:02:50.990 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 68 + 1 2023-11-17T00:02:51.025 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=69 2023-11-17T00:02:51.025 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:51.053 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:51.053 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:51.056 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:51.056 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:51.057 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:51.457 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:51.457 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:51.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:51.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:51.604 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:51.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:51.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 69 -lt 100 ']' 2023-11-17T00:02:51.605 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 69 + 1 2023-11-17T00:02:51.639 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=70 2023-11-17T00:02:51.639 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:51.669 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:51.669 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:51.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:51.673 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:51.673 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:52.032 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:52.032 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:52.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:52.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:52.172 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:52.172 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:52.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 70 -lt 100 ']' 2023-11-17T00:02:52.173 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 70 + 1 2023-11-17T00:02:52.205 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=71 2023-11-17T00:02:52.206 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:52.231 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:52.232 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:52.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:52.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:52.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:52 smithi023 ceph-mon[97824]: pgmap v422: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-17T00:02:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:52 smithi156 ceph-mon[131105]: pgmap v422: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-17T00:02:52.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:52 smithi173 ceph-mon[131534]: pgmap v422: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 15 op/s 2023-11-17T00:02:52.656 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:52.657 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:52.803 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:52.803 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:52.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:52.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:52.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 71 -lt 100 ']' 2023-11-17T00:02:52.813 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 71 + 1 2023-11-17T00:02:52.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=72 2023-11-17T00:02:52.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:52.877 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:52.878 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:52.880 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:52.881 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:52.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:53.265 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:53.266 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:53.370 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:53.370 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:53.380 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:53.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:53.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 72 -lt 100 ']' 2023-11-17T00:02:53.381 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 72 + 1 2023-11-17T00:02:53.414 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=73 2023-11-17T00:02:53.414 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:53.439 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:53.439 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:53.442 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:53.443 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:53.443 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:53.756 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:53.757 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:53.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:53.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:53.896 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:53.896 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:53.897 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 73 -lt 100 ']' 2023-11-17T00:02:53.897 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 73 + 1 2023-11-17T00:02:53.930 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=74 2023-11-17T00:02:53.930 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:53.961 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:53.961 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:53.964 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:53.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:53.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:54.339 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:54.340 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:54.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:54.445 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:54.455 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:54.456 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:54.456 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 74 -lt 100 ']' 2023-11-17T00:02:54.456 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 74 + 1 2023-11-17T00:02:54.488 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=75 2023-11-17T00:02:54.488 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:54.515 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:54.516 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:54.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:54 smithi023 ceph-mon[97824]: pgmap v423: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-11-17T00:02:54.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:54.519 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:54.519 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:54.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:54 smithi173 ceph-mon[131534]: pgmap v423: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-11-17T00:02:54.645 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:54 smithi156 ceph-mon[131105]: pgmap v423: 129 pgs: 129 active+clean; 1.3 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 13 op/s 2023-11-17T00:02:54.981 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:55.107 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:55.107 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:55.107 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:55.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:55.239 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:55.239 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 75 -lt 100 ']' 2023-11-17T00:02:55.240 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 75 + 1 2023-11-17T00:02:55.240 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=76 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:55.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:55.740 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:55.740 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:55.894 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:55.894 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:55.922 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:55.922 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:55.922 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 76 -lt 100 ']' 2023-11-17T00:02:55.923 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 76 + 1 2023-11-17T00:02:55.955 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=77 2023-11-17T00:02:55.955 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:56.042 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:56.043 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:56.045 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:56.046 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:56.047 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:56.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:56 smithi023 ceph-mon[97824]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:02:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:56 smithi173 ceph-mon[131534]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:02:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:56 smithi156 ceph-mon[131105]: pgmap v424: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:02:56.964 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:56.964 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:57.177 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:57.177 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:57.213 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:57.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:57.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 77 -lt 100 ']' 2023-11-17T00:02:57.214 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 77 + 1 2023-11-17T00:02:57.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=78 2023-11-17T00:02:57.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:57.342 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:57.342 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:57.345 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:57.346 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:57.346 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 -- 172.21.15.23:0/2092581510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54100240 msgr2=0x7f0f54100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 --2- 172.21.15.23:0/2092581510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54100240 0x7f0f54100650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009990 tx=0x7f0f44009ca0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 -- 172.21.15.23:0/2092581510 shutdown_connections 2023-11-17T00:02:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 --2- 172.21.15.23:0/2092581510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 0x7f0f54108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 --2- 172.21.15.23:0/2092581510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54100240 0x7f0f54100650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 --2- 172.21.15.23:0/2092581510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54101430 0x7f0f54101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 -- 172.21.15.23:0/2092581510 >> 172.21.15.23:0/2092581510 conn(0x7f0f540fb840 msgr2=0x7f0f540fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 -- 172.21.15.23:0/2092581510 shutdown_connections 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.584+0000 7f0f59bff700 1 -- 172.21.15.23:0/2092581510 wait complete. 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 Processor -- start 2023-11-17T00:02:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 -- start start 2023-11-17T00:02:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 0x7f0f541981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 0x7f0f5419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419fd30 con 0x7f0f54100240 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5419fe70 con 0x7f0f54101dc0 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.586+0000 7f0f59bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f541a0000 con 0x7f0f54101430 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 0x7f0f541981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:60490/0 (socket says 172.21.15.23:60490) 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3851902532 learned_addr learned my addr 172.21.15.23:0/3851902532 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f53fff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 0x7f0f5419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3851902532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 msgr2=0x7f0f5419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 0x7f0f5419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 msgr2=0x7f0f541981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 0x7f0f541981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3851902532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f440095f0 con 0x7f0f54101430 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.587+0000 7f0f537fe700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 0x7f0f541981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800e9a0 tx=0x7f0f4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f50ff9700 1 -- 172.21.15.23:0/3851902532 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48009790 con 0x7f0f54101430 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5410bba0 con 0x7f0f54101430 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f50ff9700 1 -- 172.21.15.23:0/3851902532 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f48010430 con 0x7f0f54101430 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f50ff9700 1 -- 172.21.15.23:0/3851902532 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48018680 con 0x7f0f54101430 2023-11-17T00:02:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.588+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5410ba10 con 0x7f0f54101430 2023-11-17T00:02:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.591+0000 7f0f50ff9700 1 -- 172.21.15.23:0/3851902532 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f480105a0 con 0x7f0f54101430 2023-11-17T00:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.593+0000 7f0f50ff9700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 0x7f0f3c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.593+0000 7f0f50ff9700 1 -- 172.21.15.23:0/3851902532 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0f48014070 con 0x7f0f54101430 2023-11-17T00:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.593+0000 7f0f537fe700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 0x7f0f3c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.594+0000 7f0f537fe700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 0x7f0f3c0a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400b2e0 tx=0x7f0f44005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.673+0000 7f0f6d4c4b80 1 Processor -- start 2023-11-17T00:02:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.673+0000 7f0f6d4c4b80 1 -- start start 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6579870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e657a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e647f000 con 0x5581e657ab10 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e64d1840 con 0x5581e63d56b0 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e64cacd0 con 0x5581e6579db0 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6579870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e657a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57850/0 (socket says 172.21.15.23:57850) 2023-11-17T00:02:57.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.674+0000 7f0f53fff700 1 -- 172.21.15.23:0/3776059088 learned_addr learned my addr 172.21.15.23:0/3776059088 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 -- 172.21.15.23:0/3776059088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 msgr2=0x5581e657a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e657a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 -- 172.21.15.23:0/3776059088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 msgr2=0x5581e6579870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6579870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 -- 172.21.15.23:0/3776059088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e634c5d0 con 0x5581e657ab10 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e657a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.675+0000 7f0f53fff700 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011680 tx=0x7f0f4c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a215d17204a58f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.676+0000 7f0f39ffb700 1 -- 172.21.15.23:0/3776059088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c00e040 con 0x5581e657ab10 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.676+0000 7f0f39ffb700 1 -- 172.21.15.23:0/3776059088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f4c00f040 con 0x5581e657ab10 2023-11-17T00:02:57.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.676+0000 7f0f39ffb700 1 -- 172.21.15.23:0/3776059088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c004860 con 0x5581e657ab10 2023-11-17T00:02:57.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.676+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3776059088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 msgr2=0x5581e657af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011680 tx=0x7f0f4c009920 comp rx=0 tx=0).stop 2023-11-17T00:02:57.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3776059088 shutdown_connections 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e657af80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6579870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3776059088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e657a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3776059088 >> 172.21.15.23:0/3776059088 conn(0x5581e656ef50 msgr2=0x5581e6574100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3776059088 shutdown_connections 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.677+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3776059088 wait complete. 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.678+0000 7f0f6d4c4b80 1 -- start start 2023-11-17T00:02:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.678+0000 7f0f6d4c4b80 1 Processor -- start 2023-11-17T00:02:57.681 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.681 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 0x5581e6583160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.681 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e658a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e646c880 con 0x5581e657ab10 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e646c520 con 0x5581e6579db0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e634e8c0 con 0x5581e63d56b0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f6d4c4b80 10 client.0 _ll_register_callbacks cb 0x5581e648a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e658a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 0x5581e6583160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:60500/0 (socket says 172.21.15.23:60500) 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.679+0000 7f0f537fe700 1 -- 172.21.15.23:0/1559155108 learned_addr learned my addr 172.21.15.23:0/1559155108 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 -- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 msgr2=0x5581e6583160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 0x5581e6583160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 -- 172.21.15.23:0/1559155108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 msgr2=0x5581e658a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e658a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 -- 172.21.15.23:0/1559155108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c011330 con 0x5581e63d56b0 2023-11-17T00:02:57.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 0x5581e6583160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:57.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f537fe700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0f54101290 tx=0x7f0f44009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb53e80715ffb603 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.680+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4400f670 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581e656c710 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0f4400f7d0 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4400f990 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f6d4c4b80 10 client.25504 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5581e6491130 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.681+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5581e656c710 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f4402f070 con 0x5581e63d56b0 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 10 client.25504 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 10 client.25504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 20 client.25504 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e658ea50 0x5581e6590e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 4 client.25504 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:02:57.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581e648d040 con 0x5581e658ea50 2023-11-17T00:02:57.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.682+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e658ea50 0x5581e6590e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.683+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e658ea50 0x5581e6590e60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77c4c085cd65fbd5 server_cookie=1cf3c5931c91f67d in_seq=0 out_seq=0 2023-11-17T00:02:57.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.683+0000 7f0f32ffd700 10 client.25504 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.683+0000 7f0f32ffd700 1 -- 172.21.15.23:0/1559155108 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5581e648d040 con 0x5581e658ea50 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.683+0000 7f0f32ffd700 10 client.25504 handle_command_reply: tid=0 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.683+0000 7f0f6d4c4b80 1 client.25504 shutdown 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 msgr2=0x5581e6582c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0f54101290 tx=0x7f0f44009ca0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 shutdown_connections 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e658ea50 0x5581e6590e60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e658a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e6582c20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/1559155108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e6579db0 0x5581e6583160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f32ffd700 0 client.25504 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 >> 172.21.15.23:0/1559155108 conn(0x5581e656ef50 msgr2=0x5581e65737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 shutdown_connections 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/1559155108 wait complete. 2023-11-17T00:02:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.684+0000 7f0f6d4c4b80 20 client.25504 trim_cache size 0 max 16384 2023-11-17T00:02:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.701+0000 7f0f6d4c4b80 1 Processor -- start 2023-11-17T00:02:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.701+0000 7f0f6d4c4b80 1 -- start start 2023-11-17T00:02:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.701+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e657aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.701+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e6579db0 0x5581e658a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 0x5581e657d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e657a420 con 0x5581e6579db0 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e6575930 con 0x5581e657ab10 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e657b420 con 0x5581e63d56b0 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e657aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 0x5581e657d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e657aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:60510/0 (socket says 172.21.15.23:60510) 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f537fe700 1 -- 172.21.15.23:0/4148775052 learned_addr learned my addr 172.21.15.23:0/4148775052 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e6579db0 0x5581e658a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 -- 172.21.15.23:0/4148775052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 msgr2=0x5581e657aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e657aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 -- 172.21.15.23:0/4148775052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e6579db0 msgr2=0x5581e658a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e6579db0 0x5581e658a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.702+0000 7f0f53fff700 1 -- 172.21.15.23:0/4148775052 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e6490e80 con 0x5581e657ab10 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.703+0000 7f0f53fff700 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 0x5581e657d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011300 tx=0x7f0f4c0098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=922791daff159bc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.703+0000 7f0f3a7fc700 1 -- 172.21.15.23:0/4148775052 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c00e050 con 0x5581e657ab10 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.703+0000 7f0f3a7fc700 1 -- 172.21.15.23:0/4148775052 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f4c003ec0 con 0x5581e657ab10 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.703+0000 7f0f3a7fc700 1 -- 172.21.15.23:0/4148775052 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c007b50 con 0x5581e657ab10 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/4148775052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 msgr2=0x5581e657d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 0x5581e657d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011300 tx=0x7f0f4c0098a0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/4148775052 shutdown_connections 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e657ab10 0x5581e657d980 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e63d56b0 0x5581e657aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/4148775052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e6579db0 0x5581e658a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/4148775052 >> 172.21.15.23:0/4148775052 conn(0x5581e656ef50 msgr2=0x5581e65957c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/4148775052 shutdown_connections 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/4148775052 wait complete. 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.704+0000 7f0f6d4c4b80 1 -- start start 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 Processor -- start 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6585ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e6589450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e6589990 con 0x5581e657ab10 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e6589b00 con 0x5581e63d56b0 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e6589cc0 con 0x5581e6579db0 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.705+0000 7f0f6d4c4b80 10 client.0 _ll_register_callbacks cb 0x7f0f44005730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:02:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e6589450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f537fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6585ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:60512/0 (socket says 172.21.15.23:60512) 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3943495422 learned_addr learned my addr 172.21.15.23:0/3943495422 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 msgr2=0x5581e6585ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6585ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3943495422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 msgr2=0x5581e6589450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e6589450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 -- 172.21.15.23:0/3943495422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581e634c5d0 con 0x5581e6579db0 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f53fff700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e6589450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f52ffd700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f48096900 tx=0x7f0f480ca4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c4b04f11161b10 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:02:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.706+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480b8300 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0f48010880 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f4c00f040 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f480ca120 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 10 client.25510 fetch_fsmap learned FSMap version 24 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5581e62e1c00 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f48063020 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f480b8020 con 0x5581e6579db0 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 10 client.25510 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 20 client.25510 populate_metadata read hostname 'smithi023' 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e65e3890 0x5581e65e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:02:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 4 client.25510 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:02:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.707+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5581e6499600 con 0x5581e65e3890 2023-11-17T00:02:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.708+0000 7f0f537fe700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e65e3890 0x5581e65e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:02:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.708+0000 7f0f537fe700 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e65e3890 0x5581e65e5ca0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77eccb6356887885 server_cookie=e60e4a3950205d97 in_seq=0 out_seq=0 2023-11-17T00:02:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.708+0000 7f0f32ffd700 10 client.25510 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.729+0000 7f0f32ffd700 1 -- 172.21.15.23:0/3943495422 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5581e6499600 con 0x5581e65e3890 2023-11-17T00:02:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.729+0000 7f0f32ffd700 10 client.25510 handle_command_reply: tid=0 2023-11-17T00:02:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.729+0000 7f0f6d4c4b80 1 client.25510 shutdown 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 msgr2=0x5581e6586430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f48096900 tx=0x7f0f480ca4a0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 shutdown_connections 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5581e65e3890 0x5581e65e5ca0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581e657ab10 0x5581e6589450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5581e63d56b0 0x5581e6585ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f32ffd700 0 client.25510 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:02:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 --2- 172.21.15.23:0/3943495422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5581e6579db0 0x5581e6586430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 >> 172.21.15.23:0/3943495422 conn(0x5581e656ef50 msgr2=0x5581e657d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 shutdown_connections 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 1 -- 172.21.15.23:0/3943495422 wait complete. 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.730+0000 7f0f6d4c4b80 20 client.25510 trim_cache size 0 max 16384 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:02:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 msgr2=0x7f0f3c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 0x7f0f3c0a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400b2e0 tx=0x7f0f44005fb0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 msgr2=0x7f0f541986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800e9a0 tx=0x7f0f4800ecb0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 shutdown_connections 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f54101dc0 0x7f0f5419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f54100240 0x7f0f541981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0f3c0a1150 0x7f0f3c0a3600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 --2- 172.21.15.23:0/3851902532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f54101430 0x7f0f541986f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:02:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 >> 172.21.15.23:0/3851902532 conn(0x7f0f540fb840 msgr2=0x7f0f54106d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:02:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 shutdown_connections 2023-11-17T00:02:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:02:57.731+0000 7f0f59bff700 1 -- 172.21.15.23:0/3851902532 wait complete. 2023-11-17T00:02:58.105 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:58.106 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:58.261 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:58.261 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:58.273 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:58.273 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:58.273 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 78 -lt 100 ']' 2023-11-17T00:02:58.273 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 78 + 1 2023-11-17T00:02:58.315 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=79 2023-11-17T00:02:58.315 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:58.334 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:58.334 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:58.337 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:58.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:58.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:58.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:58 smithi023 ceph-mon[97824]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-11-17T00:02:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:58 smithi173 ceph-mon[131534]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-11-17T00:02:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:58 smithi156 ceph-mon[131105]: pgmap v425: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-11-17T00:02:58.730 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:58.731 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:58.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:58.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:58.845 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:58.845 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:58.845 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 79 -lt 100 ']' 2023-11-17T00:02:58.846 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 79 + 1 2023-11-17T00:02:58.896 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=80 2023-11-17T00:02:58.896 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:58.945 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:58.946 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:58.948 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:58.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:58.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:59.339 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:59.339 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:59.467 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:02:59.467 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:02:59.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:59.481 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:02:59.481 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 80 -lt 100 ']' 2023-11-17T00:02:59.481 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 80 + 1 2023-11-17T00:02:59.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=81 2023-11-17T00:02:59.516 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:02:59.560 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:59.561 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:02:59.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:02:59.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:02:59.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:02:59.905 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:02:59.905 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:00.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:00.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:00.070 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:00.071 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:00.071 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 81 -lt 100 ']' 2023-11-17T00:03:00.071 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 81 + 1 2023-11-17T00:03:00.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:02:59 smithi173 ceph-mon[131534]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-11-17T00:03:00.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:02:59 smithi156 ceph-mon[131105]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-11-17T00:03:00.107 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=82 2023-11-17T00:03:00.107 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:00.135 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:00.136 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:00.138 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:00.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:00.140 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:00.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:02:59 smithi023 ceph-mon[97824]: pgmap v426: 129 pgs: 129 active+clean; 1.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-11-17T00:03:00.538 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:00.539 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:00.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:00.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:00.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:00.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:00.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 82 -lt 100 ']' 2023-11-17T00:03:00.681 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 82 + 1 2023-11-17T00:03:00.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=83 2023-11-17T00:03:00.714 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:00.738 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:00.739 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:00.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:00.742 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:00.742 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:01.172 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:01.173 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:01.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:01.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:01.454 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:01.454 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:01.454 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 83 -lt 100 ']' 2023-11-17T00:03:01.455 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 83 + 1 2023-11-17T00:03:01.487 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=84 2023-11-17T00:03:01.487 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:01.550 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:01.550 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:01.553 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:01.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:01.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:02.005 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:02.005 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:02.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:02.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:02.151 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:02.151 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:02.151 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 84 -lt 100 ']' 2023-11-17T00:03:02.152 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 84 + 1 2023-11-17T00:03:02.179 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=85 2023-11-17T00:03:02.179 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:02.210 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:02.211 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:02.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:02.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:02.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:02.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:02 smithi023 ceph-mon[97824]: pgmap v427: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 13 op/s 2023-11-17T00:03:02.580 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:02.580 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:02.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:02 smithi173 ceph-mon[131534]: pgmap v427: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 13 op/s 2023-11-17T00:03:02.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:02 smithi156 ceph-mon[131105]: pgmap v427: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 13 op/s 2023-11-17T00:03:02.690 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:02.690 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:02.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:02.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:02.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 85 -lt 100 ']' 2023-11-17T00:03:02.704 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 85 + 1 2023-11-17T00:03:02.745 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=86 2023-11-17T00:03:02.745 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:02.771 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:02.772 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:02.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:02.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:02.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:03.322 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:03.322 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:03.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:03.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:03.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:03.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:03.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 86 -lt 100 ']' 2023-11-17T00:03:03.472 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 86 + 1 2023-11-17T00:03:03.504 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=87 2023-11-17T00:03:03.504 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:03.535 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:03.535 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:03.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:03.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:03.539 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:03.871 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:03.872 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:03.976 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:03.976 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:03.986 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:03.986 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:03.986 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 87 -lt 100 ']' 2023-11-17T00:03:03.987 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 87 + 1 2023-11-17T00:03:04.020 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=88 2023-11-17T00:03:04.020 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:04.061 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:04.061 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:04.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:04.065 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:04.065 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:04.371 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:04.372 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:04.517 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:04.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:04.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:04 smithi023 ceph-mon[97824]: pgmap v428: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2023-11-17T00:03:04.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:04.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:04.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 88 -lt 100 ']' 2023-11-17T00:03:04.530 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 88 + 1 2023-11-17T00:03:04.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=89 2023-11-17T00:03:04.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:04.593 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:04.594 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:04.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:04.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:04.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:04.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:04 smithi173 ceph-mon[131534]: pgmap v428: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2023-11-17T00:03:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:04 smithi156 ceph-mon[131105]: pgmap v428: 129 pgs: 129 active+clean; 1.5 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2023-11-17T00:03:04.980 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:04.980 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:05.118 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:05.118 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:05.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:05.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:05.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 89 -lt 100 ']' 2023-11-17T00:03:05.130 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 89 + 1 2023-11-17T00:03:05.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=90 2023-11-17T00:03:05.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:05.226 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:05.373 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:05.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:05.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:05.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:05.621 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:05.621 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:05.759 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:05.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:05.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:05.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 90 -lt 100 ']' 2023-11-17T00:03:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 90 + 1 2023-11-17T00:03:05.803 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=91 2023-11-17T00:03:05.803 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:05.834 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:05.834 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:05.837 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:05.837 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:05.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:06.179 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:06.180 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:06.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:06.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:06.311 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:06.311 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:06.311 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 91 -lt 100 ']' 2023-11-17T00:03:06.312 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 91 + 1 2023-11-17T00:03:06.348 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=92 2023-11-17T00:03:06.348 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:06.375 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:06.376 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:06.378 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:06.378 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:06.379 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:06 smithi023 ceph-mon[97824]: pgmap v429: 129 pgs: 129 active+clean; 1.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:03:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:06 smithi173 ceph-mon[131534]: pgmap v429: 129 pgs: 129 active+clean; 1.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:03:06.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:06 smithi156 ceph-mon[131105]: pgmap v429: 129 pgs: 129 active+clean; 1.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 16 op/s 2023-11-17T00:03:06.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:06.721 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:06.850 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:06.851 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:06.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:06.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:06.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 92 -lt 100 ']' 2023-11-17T00:03:06.862 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 92 + 1 2023-11-17T00:03:06.911 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=93 2023-11-17T00:03:06.912 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:06.950 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:06.950 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:06.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:06.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:06.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:07.295 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:07.296 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:07.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:07.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:07.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:07.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:07.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 93 -lt 100 ']' 2023-11-17T00:03:07.564 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 93 + 1 2023-11-17T00:03:07.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=94 2023-11-17T00:03:07.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:07.625 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:07.626 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:07.628 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:07.628 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:07.629 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:08.245 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:08.246 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:08.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:08.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:08.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:08.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:08.403 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 94 -lt 100 ']' 2023-11-17T00:03:08.404 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 94 + 1 2023-11-17T00:03:08.436 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=95 2023-11-17T00:03:08.436 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:08.467 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:08.467 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:08.470 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:08.470 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:08.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:08.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:08 smithi023 ceph-mon[97824]: pgmap v430: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:08 smithi173 ceph-mon[131534]: pgmap v430: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:08 smithi156 ceph-mon[131105]: pgmap v430: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:08.813 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:08.813 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:08.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:08.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:08.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:08.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:08.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 95 -lt 100 ']' 2023-11-17T00:03:08.963 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 95 + 1 2023-11-17T00:03:08.997 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=96 2023-11-17T00:03:08.997 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:09.026 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:09.026 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:09.028 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:09.029 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:09.029 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:09.422 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:09.423 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:09.609 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:09.609 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:09.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:09.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:09.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 96 -lt 100 ']' 2023-11-17T00:03:09.620 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 96 + 1 2023-11-17T00:03:09.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=97 2023-11-17T00:03:09.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:09.683 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:09.684 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:09.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:09.687 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:09.687 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:10.070 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:10.071 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:10.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:09 smithi156 ceph-mon[131105]: pgmap v431: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:10.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:09 smithi173 ceph-mon[131534]: pgmap v431: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:10.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:10.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:10.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:10.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:10.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 97 -lt 100 ']' 2023-11-17T00:03:10.186 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 97 + 1 2023-11-17T00:03:10.219 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=98 2023-11-17T00:03:10.219 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:10.245 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:10.245 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:10.247 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:10.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:10.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:10.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:09 smithi023 ceph-mon[97824]: pgmap v431: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 14 op/s 2023-11-17T00:03:10.562 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:10.562 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:10.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:10.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:10.704 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:10.704 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:10.705 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 98 -lt 100 ']' 2023-11-17T00:03:10.705 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 98 + 1 2023-11-17T00:03:10.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=99 2023-11-17T00:03:10.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:10.766 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:10.767 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:10.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:10.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:10.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:11.221 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:11.221 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:11.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:11.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:11.435 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:11.435 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:11.435 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 99 -lt 100 ']' 2023-11-17T00:03:11.436 INFO:tasks.workunit.client.0.smithi023.stderr:++ expr 99 + 1 2023-11-17T00:03:11.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ c=100 2023-11-17T00:03:11.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:11.545 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:11.545 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:11.548 INFO:tasks.workunit.client.0.smithi023.stderr:+ c1=2 2023-11-17T00:03:11.548 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-11-17T00:03:11.549 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-17T00:03:12.328 INFO:tasks.workunit.client.0.smithi023.stderr:++ getfacl d1 2023-11-17T00:03:12.329 INFO:tasks.workunit.client.0.smithi023.stderr:++ grep -c nobody:rw 2023-11-17T00:03:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:12 smithi023 ceph-mon[97824]: pgmap v432: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 13 op/s 2023-11-17T00:03:12.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ c2=2 2023-11-17T00:03:12.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:12.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:12.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -ne 2 ']' 2023-11-17T00:03:12.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 100 -lt 100 ']' 2023-11-17T00:03:12.600 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1 2023-11-17T00:03:12.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:12 smithi173 ceph-mon[131534]: pgmap v432: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 13 op/s 2023-11-17T00:03:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:12 smithi156 ceph-mon[131105]: pgmap v432: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 13 op/s 2023-11-17T00:03:12.652 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-11-17T00:03:12.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-11-17T00:03:12.665 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-11-17T00:03:12.666 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n system.posix_acl_access d1 2023-11-17T00:03:12.681 INFO:tasks.workunit.client.0.smithi023.stderr:d1: system.posix_acl_access: No such attribute 2023-11-17T00:03:12.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-11-17T00:03:12.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-11-17T00:03:12.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n system.posix_acl_default . 2023-11-17T00:03:12.683 INFO:tasks.workunit.client.0.smithi023.stderr:.: system.posix_acl_default: No such attribute 2023-11-17T00:03:12.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-11-17T00:03:12.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1 2023-11-17T00:03:12.688 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .. 2023-11-17T00:03:12.688 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir testdir 2023-11-17T00:03:12.700 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-11-17T00:03:12.701 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-11-17T00:03:12.701 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:03:12.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:03:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 -- 172.21.15.23:0/2821308329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc101de0 msgr2=0x7f4bfc1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 --2- 172.21.15.23:0/2821308329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc101de0 0x7f4bfc1086a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400b0a0 tx=0x7f4bf400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 -- 172.21.15.23:0/2821308329 shutdown_connections 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 --2- 172.21.15.23:0/2821308329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc101de0 0x7f4bfc1086a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 --2- 172.21.15.23:0/2821308329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc100260 0x7f4bfc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 --2- 172.21.15.23:0/2821308329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101450 0x7f4bfc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 -- 172.21.15.23:0/2821308329 >> 172.21.15.23:0/2821308329 conn(0x7f4bfc0fb880 msgr2=0x7f4bfc0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 -- 172.21.15.23:0/2821308329 shutdown_connections 2023-11-17T00:03:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.906+0000 7f4c02dff700 1 -- 172.21.15.23:0/2821308329 wait complete. 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 Processor -- start 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 -- start start 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 0x7f4bfc196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101de0 0x7f4bfc19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc19dc40 con 0x7f4bfc100260 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc19dd80 con 0x7f4bfc101de0 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c02dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc19df10 con 0x7f4bfc101450 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59928/0 (socket says 172.21.15.23:59928) 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3460661253 learned_addr learned my addr 172.21.15.23:0/3460661253 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.908+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 0x7f4bfc196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c0139c700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101de0 0x7f4bfc19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3460661253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 msgr2=0x7f4bfc196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 0x7f4bfc196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3460661253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101de0 msgr2=0x7f4bfc19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101de0 0x7f4bfc19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3460661253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf4009d00 con 0x7f4bfc100260 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 0x7f4bfc196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000e970 tx=0x7f4bf000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4bf9ffb700 1 -- 172.21.15.23:0/3460661253 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0009790 con 0x7f4bfc100260 2023-11-17T00:03:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4bf9ffb700 1 -- 172.21.15.23:0/3460661253 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bf000c7e0 con 0x7f4bfc100260 2023-11-17T00:03:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc10bc40 con 0x7f4bfc100260 2023-11-17T00:03:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.909+0000 7f4bf9ffb700 1 -- 172.21.15.23:0/3460661253 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0003c70 con 0x7f4bfc100260 2023-11-17T00:03:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.910+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc10bab0 con 0x7f4bfc100260 2023-11-17T00:03:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.913+0000 7f4bf9ffb700 1 -- 172.21.15.23:0/3460661253 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4bf000cc50 con 0x7f4bfc100260 2023-11-17T00:03:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.914+0000 7f4bf9ffb700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 0x7f4bec0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.914+0000 7f4bf9ffb700 1 -- 172.21.15.23:0/3460661253 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4bf0014070 con 0x7f4bfc100260 2023-11-17T00:03:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.914+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 0x7f4bec0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:12.915+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 0x7f4bec0a3680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4be8011490 tx=0x7f4be8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:13.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.075+0000 7f4c16654b80 1 Processor -- start 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.075+0000 7f4c16654b80 1 -- start start 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c8419039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c841904390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c841804000 con 0x55c84175a6b0 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c841856840 con 0x55c8419031c0 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c16654b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84184fcd0 con 0x55c841903f20 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59942/0 (socket says 172.21.15.23:59942) 2023-11-17T00:03:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3306813738 learned_addr learned my addr 172.21.15.23:0/3306813738 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:13.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c8419039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c0139c700 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c841904390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3306813738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 msgr2=0x55c841904390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c841904390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3306813738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 msgr2=0x55c8419039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c8419039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.076+0000 7f4c00b9b700 1 -- 172.21.15.23:0/3306813738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8416d15d0 con 0x55c84175a6b0 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.077+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00968d0 tx=0x7f4bf0003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1be637eedf5d549 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.077+0000 7f4be6ffd700 1 -- 172.21.15.23:0/3306813738 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf009de90 con 0x55c84175a6b0 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.077+0000 7f4be6ffd700 1 -- 172.21.15.23:0/3306813738 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bf00a22b0 con 0x55c84175a6b0 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.077+0000 7f4be6ffd700 1 -- 172.21.15.23:0/3306813738 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0099350 con 0x55c84175a6b0 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- 172.21.15.23:0/3306813738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 msgr2=0x55c841902c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00968d0 tx=0x7f4bf0003710 comp rx=0 tx=0).stop 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- 172.21.15.23:0/3306813738 shutdown_connections 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c841904390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c841902c80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 --2- 172.21.15.23:0/3306813738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c8419039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- 172.21.15.23:0/3306813738 >> 172.21.15.23:0/3306813738 conn(0x55c8418f3f50 msgr2=0x55c8418f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- 172.21.15.23:0/3306813738 shutdown_connections 2023-11-17T00:03:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- 172.21.15.23:0/3306813738 wait complete. 2023-11-17T00:03:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.078+0000 7f4c16654b80 1 -- start start 2023-11-17T00:03:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 Processor -- start 2023-11-17T00:03:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8417f1880 con 0x55c841903f20 2023-11-17T00:03:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8417f1520 con 0x55c8419031c0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8416d38c0 con 0x55c84175a6b0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.079+0000 7f4c16654b80 10 client.0 _ll_register_callbacks cb 0x55c84180f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c0139c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4bfbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c0139c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59954/0 (socket says 172.21.15.23:59954) 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c0139c700 1 -- 172.21.15.23:0/1827008311 learned_addr learned my addr 172.21.15.23:0/1827008311 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 -- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 msgr2=0x55c84190c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 -- 172.21.15.23:0/1827008311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 msgr2=0x55c841913600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 -- 172.21.15.23:0/1827008311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc10bab0 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c0139c700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190c080 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0005950 tx=0x7f4bf009de90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9efcbf8fcb187e40 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.080+0000 7f4bdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00982c0 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4bf009a210 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8418f1710 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4bf00ca270 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 10 client.25522 fetch_fsmap learned FSMap version 24 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c841816130 con 0x55c84175a6b0 2023-11-17T00:03:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8418f1710 con 0x55c84175a6b0 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4bf0095c50 con 0x55c84175a6b0 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 10 client.25522 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 10 client.25522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 20 client.25522 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c841917b70 0x55c841919f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 4 client.25522 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.081+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c841812040 con 0x55c841917b70 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.082+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c841917b70 0x55c841919f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.082+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c841917b70 0x55c841919f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a5d339c105a7e42 server_cookie=cc8143bbc309e8f5 in_seq=0 out_seq=0 2023-11-17T00:03:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.082+0000 7f4bdbfff700 10 client.25522 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.083+0000 7f4bdbfff700 1 -- 172.21.15.23:0/1827008311 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c841812040 con 0x55c841917b70 2023-11-17T00:03:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.083+0000 7f4bdbfff700 10 client.25522 handle_command_reply: tid=0 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.083+0000 7f4c16654b80 1 client.25522 shutdown 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.083+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 msgr2=0x55c84190c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.083+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190c080 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0005950 tx=0x7f4bf009de90 comp rx=0 tx=0).stop 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 shutdown_connections 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c841917b70 0x55c841919f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c841903f20 0x55c841913600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190c080 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 --2- 172.21.15.23:0/1827008311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 >> 172.21.15.23:0/1827008311 conn(0x55c8418f3f50 msgr2=0x55c8418f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4bdbfff700 0 client.25522 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 shutdown_connections 2023-11-17T00:03:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 1 -- 172.21.15.23:0/1827008311 wait complete. 2023-11-17T00:03:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.084+0000 7f4c16654b80 20 client.25522 trim_cache size 0 max 16384 2023-11-17T00:03:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 Processor -- start 2023-11-17T00:03:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 -- start start 2023-11-17T00:03:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c84190a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c8419135c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84190bc50 con 0x55c84175a6b0 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8418f1280 con 0x55c8419031c0 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c16654b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8418fa930 con 0x55c841903f20 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c84190a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4bfbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4bfbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:41134/0 (socket says 172.21.15.23:41134) 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.100+0000 7f4bfbfff700 1 -- 172.21.15.23:0/4096720107 learned_addr learned my addr 172.21.15.23:0/4096720107 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4c0139c700 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c8419135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 -- 172.21.15.23:0/4096720107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 msgr2=0x55c8419135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c8419135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 -- 172.21.15.23:0/4096720107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 msgr2=0x55c84190a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c84190a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 -- 172.21.15.23:0/4096720107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c841815e80 con 0x55c8419031c0 2023-11-17T00:03:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c84190a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4be8013010 tx=0x7f4be8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8059c8bc0bae898b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4be77fe700 1 -- 172.21.15.23:0/4096720107 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8024040 con 0x55c8419031c0 2023-11-17T00:03:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4be77fe700 1 -- 172.21.15.23:0/4096720107 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4be8003da0 con 0x55c8419031c0 2023-11-17T00:03:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.101+0000 7f4be77fe700 1 -- 172.21.15.23:0/4096720107 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be801ea80 con 0x55c8419031c0 2023-11-17T00:03:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 -- 172.21.15.23:0/4096720107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 msgr2=0x55c84190a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4be8013010 tx=0x7f4be8014000 comp rx=0 tx=0).stop 2023-11-17T00:03:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 -- 172.21.15.23:0/4096720107 shutdown_connections 2023-11-17T00:03:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c841903f20 0x55c8419135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c84175a6b0 0x55c84190a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 --2- 172.21.15.23:0/4096720107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8419031c0 0x55c84190a750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 -- 172.21.15.23:0/4096720107 >> 172.21.15.23:0/4096720107 conn(0x55c8418f3f50 msgr2=0x55c84191e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 -- 172.21.15.23:0/4096720107 shutdown_connections 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.102+0000 7f4c16654b80 1 -- 172.21.15.23:0/4096720107 wait complete. 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.103+0000 7f4c16654b80 1 -- start start 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.103+0000 7f4c16654b80 1 Processor -- start 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.103+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.103+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8419031c0 0x55c84190fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c16654b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c841903f20 0x55c841912790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c16654b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c841912cd0 con 0x55c8419031c0 2023-11-17T00:03:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c16654b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c841912e40 con 0x55c841903f20 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c16654b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c841913000 con 0x55c84175a6b0 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c16654b80 10 client.0 _ll_register_callbacks cb 0x7f4be8004030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4bfbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8419031c0 0x55c84190fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:55720/0 (socket says 172.21.15.23:55720) 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 -- 172.21.15.23:0/270132045 learned_addr learned my addr 172.21.15.23:0/270132045 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c0139c700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c841903f20 0x55c841912790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 -- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c841903f20 msgr2=0x55c841912790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c841903f20 0x55c841912790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 -- 172.21.15.23:0/270132045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8419031c0 msgr2=0x55c84190fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8419031c0 0x55c84190fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 -- 172.21.15.23:0/270132045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c841816130 con 0x55c84175a6b0 2023-11-17T00:03:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4c00b9b700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0095fb0 tx=0x7f4bf000c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f725dfbbd5098f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.104+0000 7f4bdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:03:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00add60 con 0x55c84175a6b0 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4bf00982c0 con 0x55c84175a6b0 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4bf00a22b0 con 0x55c84175a6b0 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4bf0003dd0 con 0x55c84175a6b0 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 10 client.25528 fetch_fsmap learned FSMap version 24 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f4be8017070 con 0x55c84175a6b0 2023-11-17T00:03:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bf00a22b0 con 0x55c84175a6b0 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4bf0098dc0 con 0x55c84175a6b0 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 10 client.25528 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 10 client.25528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 20 client.25528 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.105+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c84196c2d0 0x55c84196e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.106+0000 7f4c16654b80 4 client.25528 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:03:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.106+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c84181e600 con 0x55c84196c2d0 2023-11-17T00:03:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.106+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c84196c2d0 0x55c84196e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.106+0000 7f4bdbfff700 10 client.25528 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.106+0000 7f4bfbfff700 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c84196c2d0 0x55c84196e6b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdaf6c1519081512 server_cookie=6c39abf24baa4135 in_seq=0 out_seq=0 2023-11-17T00:03:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4bdbfff700 1 -- 172.21.15.23:0/270132045 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c84181e600 con 0x55c84196c2d0 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4bdbfff700 10 client.25528 handle_command_reply: tid=0 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 client.25528 shutdown 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 msgr2=0x55c84190f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0095fb0 tx=0x7f4bf000c4a0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 shutdown_connections 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c841903f20 0x55c841912790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c84175a6b0 0x55c84190f750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c84196c2d0 0x55c84196e6b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 --2- 172.21.15.23:0/270132045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8419031c0 0x55c84190fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4bdbfff700 0 client.25528 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 >> 172.21.15.23:0/270132045 conn(0x55c8418f3f50 msgr2=0x55c841906d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 shutdown_connections 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.127+0000 7f4c16654b80 1 -- 172.21.15.23:0/270132045 wait complete. 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c16654b80 20 client.25528 trim_cache size 0 max 16384 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:03:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:03:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:03:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:03:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 msgr2=0x7f4bec0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 0x7f4bec0a3680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4be8011490 tx=0x7f4be8009260 comp rx=0 tx=0).stop 2023-11-17T00:03:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 msgr2=0x7f4bfc1960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000e970 tx=0x7f4bf000ec80 comp rx=0 tx=0).stop 2023-11-17T00:03:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 shutdown_connections 2023-11-17T00:03:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bfc101de0 0x7f4bfc19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bfc100260 0x7f4bfc1960c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4bec0a11d0 0x7f4bec0a3680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 --2- 172.21.15.23:0/3460661253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bfc101450 0x7f4bfc196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 >> 172.21.15.23:0/3460661253 conn(0x7f4bfc0fb880 msgr2=0x7f4bfc104840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 shutdown_connections 2023-11-17T00:03:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:13.128+0000 7f4c02dff700 1 -- 172.21.15.23:0/3460661253 wait complete. 2023-11-17T00:03:13.357 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-11-17T00:03:13.358 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-11-17T00:03:13.503 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-11-17T00:03:13.503 INFO:tasks.workunit.client.0.smithi023.stderr:+ file=test_chmod.150050 2023-11-17T00:03:13.503 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo foo 2023-11-17T00:03:13.513 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.513 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 400 test_chmod.150050 2023-11-17T00:03:13.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ check_perms test_chmod.150050 -r-------- 2023-11-17T00:03:13.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ file=test_chmod.150050 2023-11-17T00:03:13.515 INFO:tasks.workunit.client.0.smithi023.stderr:++ ls -la test_chmod.150050 2023-11-17T00:03:13.527 INFO:tasks.workunit.client.0.smithi023.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Nov 17 00:03 test_chmod.150050' 2023-11-17T00:03:13.527 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.527 INFO:tasks.workunit.client.0.smithi023.stderr:+ perms=-r-------- 2023-11-17T00:03:13.530 INFO:tasks.workunit.client.0.smithi023.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 17 00:03 test_chmod.150050 2023-11-17T00:03:13.530 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-11-17T00:03:13.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ test -r-------- '!=' -r--------. 2023-11-17T00:03:13.597 INFO:tasks.workunit.client.0.smithi023.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 17 00:03 test_chmod.150050 2023-11-17T00:03:13.597 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ test -r--------. '!=' -r--------. 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ set +e 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo bar 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.150050: Permission denied 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 1 = 0 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-11-17T00:03:13.601 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 600 test_chmod.150050 2023-11-17T00:03:13.603 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo bar 2023-11-17T00:03:13.603 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.603 INFO:tasks.workunit.client.0.smithi023.stderr:+ check_perms test_chmod.150050 -rw------- 2023-11-17T00:03:13.603 INFO:tasks.workunit.client.0.smithi023.stderr:+ file=test_chmod.150050 2023-11-17T00:03:13.604 INFO:tasks.workunit.client.0.smithi023.stderr:++ ls -la test_chmod.150050 2023-11-17T00:03:13.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Nov 17 00:03 test_chmod.150050' 2023-11-17T00:03:13.607 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.607 INFO:tasks.workunit.client.0.smithi023.stderr:+ perms=-rw------- 2023-11-17T00:03:13.608 INFO:tasks.workunit.client.0.smithi023.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 17 00:03 test_chmod.150050 2023-11-17T00:03:13.608 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-11-17T00:03:13.612 INFO:tasks.workunit.client.0.smithi023.stderr:+ test -rw------- '!=' -rw-------. 2023-11-17T00:03:13.613 INFO:tasks.workunit.client.0.smithi023.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 17 00:03 test_chmod.150050 2023-11-17T00:03:13.613 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-11-17T00:03:13.617 INFO:tasks.workunit.client.0.smithi023.stderr:+ test -rw-------. '!=' -rw-------. 2023-11-17T00:03:13.617 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo foo 2023-11-17T00:03:13.617 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 0 '!=' 0 2023-11-17T00:03:13.618 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:03:13.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:03:13.710 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-11-17T00:03:13.710 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-11-17T00:03:13.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-11-17T00:03:13.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p testdir 2023-11-17T00:03:13.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ file=test_chmod.150109 2023-11-17T00:03:13.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo foo 2023-11-17T00:03:13.774 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo chmod 600 testdir 2023-11-17T00:03:13.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure cat testdir/test_chmod.150109 2023-11-17T00:03:13.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat testdir/test_chmod.150109 2023-11-17T00:03:13.830 INFO:tasks.workunit.client.0.smithi023.stderr:cat: testdir/test_chmod.150109: Permission denied 2023-11-17T00:03:13.830 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-11-17T00:03:13.830 INFO:tasks.workunit.client.0.smithi023.stderr:+ sudo cat testdir/test_chmod.150109 2023-11-17T00:03:13.856 INFO:tasks.workunit.client.0.smithi023.stdout:foo 2023-11-17T00:03:13.860 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:03:13.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:03:13.952 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-11-17T00:03:13.952 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-11-17T00:03:14.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:14 smithi023 ceph-mon[97824]: pgmap v433: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 9 op/s 2023-11-17T00:03:14.535 INFO:tasks.workunit.client.0.smithi023.stdout:ok 2023-11-17T00:03:14.542 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:03:14.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:03:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:14 smithi173 ceph-mon[131534]: pgmap v433: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 9 op/s 2023-11-17T00:03:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:14 smithi156 ceph-mon[131105]: pgmap v433: 129 pgs: 129 active+clean; 1.6 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 9 op/s 2023-11-17T00:03:14.629 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-11-17T00:03:14.629 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-11-17T00:03:14.720 INFO:tasks.workunit.client.0.smithi023.stdout:creating folder hierarchy 2023-11-17T00:03:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:16 smithi023 ceph-mon[97824]: pgmap v434: 129 pgs: 129 active+clean; 1.8 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 14 op/s 2023-11-17T00:03:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:16 smithi173 ceph-mon[131534]: pgmap v434: 129 pgs: 129 active+clean; 1.8 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 14 op/s 2023-11-17T00:03:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:16 smithi156 ceph-mon[131105]: pgmap v434: 129 pgs: 129 active+clean; 1.8 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 29 KiB/s wr, 14 op/s 2023-11-17T00:03:18.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:18 smithi023 ceph-mon[97824]: pgmap v435: 129 pgs: 129 active+clean; 1.8 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 KiB/s wr, 17 op/s 2023-11-17T00:03:18.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:18 smithi173 ceph-mon[131534]: pgmap v435: 129 pgs: 129 active+clean; 1.8 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 KiB/s wr, 17 op/s 2023-11-17T00:03:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:18 smithi156 ceph-mon[131105]: pgmap v435: 129 pgs: 129 active+clean; 1.8 MiB data, 91 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 KiB/s wr, 17 op/s 2023-11-17T00:03:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:19 smithi173 ceph-mon[131534]: pgmap v436: 129 pgs: 129 active+clean; 5.1 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 346 KiB/s wr, 23 op/s 2023-11-17T00:03:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:19 smithi156 ceph-mon[131105]: pgmap v436: 129 pgs: 129 active+clean; 5.1 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 346 KiB/s wr, 23 op/s 2023-11-17T00:03:20.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:19 smithi023 ceph-mon[97824]: pgmap v436: 129 pgs: 129 active+clean; 5.1 MiB data, 100 MiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 346 KiB/s wr, 23 op/s 2023-11-17T00:03:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:22 smithi023 ceph-mon[97824]: pgmap v437: 129 pgs: 129 active+clean; 7.9 MiB data, 103 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 593 KiB/s wr, 28 op/s 2023-11-17T00:03:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:22 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:03:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:22 smithi156 ceph-mon[131105]: pgmap v437: 129 pgs: 129 active+clean; 7.9 MiB data, 103 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 593 KiB/s wr, 28 op/s 2023-11-17T00:03:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:22 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:03:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:22 smithi173 ceph-mon[131534]: pgmap v437: 129 pgs: 129 active+clean; 7.9 MiB data, 103 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 593 KiB/s wr, 28 op/s 2023-11-17T00:03:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:22 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:03:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:24 smithi023 ceph-mon[97824]: pgmap v438: 129 pgs: 129 active+clean; 7.9 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 611 KiB/s wr, 26 op/s 2023-11-17T00:03:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:03:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:03:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:24 smithi173 ceph-mon[131534]: pgmap v438: 129 pgs: 129 active+clean; 7.9 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 611 KiB/s wr, 26 op/s 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:24 smithi156 ceph-mon[131105]: pgmap v438: 129 pgs: 129 active+clean; 7.9 MiB data, 116 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 611 KiB/s wr, 26 op/s 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:03:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:03:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:03:25.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:25 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:25 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:03:25.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:25 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:25 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:03:25.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:25 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:25 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:25 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:25 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:25 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:25 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:25 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:03:25.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:25 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 -- 172.21.15.23:0/3830557569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ffdb0 msgr2=0x7f9adc100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/3830557569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc100220 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0099a0 tx=0x7f9acc009cb0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 -- 172.21.15.23:0/3830557569 shutdown_connections 2023-11-17T00:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/3830557569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc100220 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/3830557569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc0feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/3830557569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ff400 0x7f9adc0ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 -- 172.21.15.23:0/3830557569 >> 172.21.15.23:0/3830557569 conn(0x7f9adc0fa720 msgr2=0x7f9adc0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 -- 172.21.15.23:0/3830557569 shutdown_connections 2023-11-17T00:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.346+0000 7f9ae25ff700 1 -- 172.21.15.23:0/3830557569 wait complete. 2023-11-17T00:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.348+0000 7f9ae25ff700 1 Processor -- start 2023-11-17T00:03:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.348+0000 7f9ae25ff700 1 -- start start 2023-11-17T00:03:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 0x7f9adc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc19fd40 con 0x7f9adc0ffdb0 2023-11-17T00:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc19fe80 con 0x7f9adc0ff400 2023-11-17T00:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae25ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc1a0010 con 0x7f9adc0feab0 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 0x7f9adc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 0x7f9adc198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:48302/0 (socket says 172.21.15.23:48302) 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:59694/0 (socket says 172.21.15.23:59694) 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9adb7fe700 1 -- 172.21.15.23:0/1716289726 learned_addr learned my addr 172.21.15.23:0/1716289726 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.349+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 -- 172.21.15.23:0/1716289726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 msgr2=0x7f9adc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 0x7f9adc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 -- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ffdb0 msgr2=0x7f9adc19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 -- 172.21.15.23:0/1716289726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0095d0 con 0x7f9adc0feab0 2023-11-17T00:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.350+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400ef10 tx=0x7f9ad400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.351+0000 7f9ad97fa700 1 -- 172.21.15.23:0/1716289726 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400ca70 con 0x7f9adc0feab0 2023-11-17T00:03:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.351+0000 7f9ad97fa700 1 -- 172.21.15.23:0/1716289726 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ad400f040 con 0x7f9adc0feab0 2023-11-17T00:03:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.351+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc10dde0 con 0x7f9adc0feab0 2023-11-17T00:03:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.351+0000 7f9ad97fa700 1 -- 172.21.15.23:0/1716289726 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40146e0 con 0x7f9adc0feab0 2023-11-17T00:03:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.351+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc10dc50 con 0x7f9adc0feab0 2023-11-17T00:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.356+0000 7f9ad97fa700 1 -- 172.21.15.23:0/1716289726 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9ad4007500 con 0x7f9adc0feab0 2023-11-17T00:03:26.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.358+0000 7f9ad97fa700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 0x7f9ac40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.358+0000 7f9ad97fa700 1 -- 172.21.15.23:0/1716289726 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f9ad40ccf50 con 0x7f9adc0feab0 2023-11-17T00:03:26.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.358+0000 7f9af5e5ab80 1 Processor -- start 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 0x7f9ac40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 -- start start 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61c2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda60bf000 con 0x55cda60156b0 2023-11-17T00:03:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda6111840 con 0x55cda61c3320 2023-11-17T00:03:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.359+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda610acd0 con 0x55cda61c25c0 2023-11-17T00:03:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:48306/0 (socket says 172.21.15.23:48306) 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 -- 172.21.15.23:0/545982945 learned_addr learned my addr 172.21.15.23:0/545982945 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 -- 172.21.15.23:0/545982945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 msgr2=0x55cda61c2de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61c2de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 -- 172.21.15.23:0/545982945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 msgr2=0x55cda61c2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9ae0b9c700 1 -- 172.21.15.23:0/545982945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda5f8c5d0 con 0x55cda61c3320 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.360+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 0x7f9ac40a3750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0007c90 tx=0x7f9ad0007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.362+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00b2e0 tx=0x7f9acc009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fb9c575bbc0e57e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.362+0000 7f9ac27fc700 1 -- 172.21.15.23:0/545982945 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00b860 con 0x55cda61c3320 2023-11-17T00:03:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.362+0000 7f9ac27fc700 1 -- 172.21.15.23:0/545982945 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9acc01f040 con 0x55cda61c3320 2023-11-17T00:03:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.362+0000 7f9ac27fc700 1 -- 172.21.15.23:0/545982945 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00fd30 con 0x55cda61c3320 2023-11-17T00:03:26.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.363+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/545982945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 msgr2=0x55cda61c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.363+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00b2e0 tx=0x7f9acc009510 comp rx=0 tx=0).stop 2023-11-17T00:03:26.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/545982945 shutdown_connections 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61c3790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/545982945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/545982945 >> 172.21.15.23:0/545982945 conn(0x55cda61aef50 msgr2=0x55cda61b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/545982945 shutdown_connections 2023-11-17T00:03:26.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.364+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/545982945 wait complete. 2023-11-17T00:03:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.365+0000 7f9af5e5ab80 1 -- start start 2023-11-17T00:03:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.365+0000 7f9af5e5ab80 1 Processor -- start 2023-11-17T00:03:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 0x55cda61cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda60ac880 con 0x55cda61c25c0 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda60ac520 con 0x55cda61c3320 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5f8e8c0 con 0x55cda60156b0 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9af5e5ab80 10 client.0 _ll_register_callbacks cb 0x55cda60ca720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.366+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 0x55cda61cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9ae0b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:59704/0 (socket says 172.21.15.23:59704) 2023-11-17T00:03:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 -- 172.21.15.23:0/1151236299 learned_addr learned my addr 172.21.15.23:0/1151236299 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 -- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 msgr2=0x55cda61d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 -- 172.21.15.23:0/1151236299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 msgr2=0x55cda61cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 0x55cda61cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adbfff700 1 -- 172.21.15.23:0/1151236299 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc10b240 con 0x55cda60156b0 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.367+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 0x55cda61cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9adbfff700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4099f40 tx=0x7f9ad4096400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39db914d94d3678c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9abb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4099fb0 con 0x55cda60156b0 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f9ad409dfe0 con 0x55cda60156b0 2023-11-17T00:03:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cda61ac710 con 0x55cda60156b0 2023-11-17T00:03:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.368+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ad40a2400 con 0x55cda60156b0 2023-11-17T00:03:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.369+0000 7f9af5e5ab80 10 client.25555 fetch_fsmap learned FSMap version 24 2023-11-17T00:03:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.369+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55cda60d1130 con 0x55cda60156b0 2023-11-17T00:03:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.369+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ad40c8020 con 0x55cda60156b0 2023-11-17T00:03:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.369+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9ad40cab60 con 0x55cda60156b0 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 10 client.25555 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 10 client.25555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 20 client.25555 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda61d72c0 0x55cda61d96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 4 client.25555 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cda60cd040 con 0x55cda61d72c0 2023-11-17T00:03:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.370+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda61d72c0 0x55cda61d96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.371+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda61d72c0 0x55cda61d96d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad52c81b7677be2 server_cookie=29f915d0182a6c7a in_seq=0 out_seq=0 2023-11-17T00:03:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.372+0000 7f9abb7fe700 10 client.25555 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9abb7fe700 1 -- 172.21.15.23:0/1151236299 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cda60cd040 con 0x55cda61d72c0 2023-11-17T00:03:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9abb7fe700 10 client.25555 handle_command_reply: tid=0 2023-11-17T00:03:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 client.25555 shutdown 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 msgr2=0x55cda61cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4099f40 tx=0x7f9ad4096400 comp rx=0 tx=0).stop 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 shutdown_connections 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61cb4b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda61d72c0 0x55cda61d96d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/1151236299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c25c0 0x55cda61cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9abb7fe700 0 client.25555 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:26.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.373+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 >> 172.21.15.23:0/1151236299 conn(0x55cda61aef50 msgr2=0x55cda61b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.374+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 shutdown_connections 2023-11-17T00:03:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.374+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/1151236299 wait complete. 2023-11-17T00:03:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.374+0000 7f9af5e5ab80 20 client.25555 trim_cache size 0 max 16384 2023-11-17T00:03:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 Processor -- start 2023-11-17T00:03:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 -- start start 2023-11-17T00:03:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c3320 0x55cda61c6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61aeba0 con 0x55cda61c3320 2023-11-17T00:03:26.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61d4720 con 0x55cda61c25c0 2023-11-17T00:03:26.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.396+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61b5bf0 con 0x55cda60156b0 2023-11-17T00:03:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:48328/0 (socket says 172.21.15.23:48328) 2023-11-17T00:03:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 -- 172.21.15.23:0/2303088997 learned_addr learned my addr 172.21.15.23:0/2303088997 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adbfff700 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c3320 0x55cda61c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 -- 172.21.15.23:0/2303088997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 msgr2=0x55cda61c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 -- 172.21.15.23:0/2303088997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c3320 msgr2=0x55cda61c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c3320 0x55cda61c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 -- 172.21.15.23:0/2303088997 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda60d0e80 con 0x55cda61c25c0 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.397+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0010a40 tx=0x7f9ad0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f81aedfb804b67 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.398+0000 7f9ac2ffd700 1 -- 172.21.15.23:0/2303088997 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0027520 con 0x55cda61c25c0 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.398+0000 7f9ac2ffd700 1 -- 172.21.15.23:0/2303088997 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ad0015070 con 0x55cda61c25c0 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.398+0000 7f9ac2ffd700 1 -- 172.21.15.23:0/2303088997 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad001f890 con 0x55cda61c25c0 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/2303088997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 msgr2=0x55cda61d29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0010a40 tx=0x7f9ad0013000 comp rx=0 tx=0).stop 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/2303088997 shutdown_connections 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda61c3320 0x55cda61c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda60156b0 0x55cda61c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/2303088997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c25c0 0x55cda61d29f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/2303088997 >> 172.21.15.23:0/2303088997 conn(0x55cda61aef50 msgr2=0x55cda61de030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/2303088997 shutdown_connections 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.399+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/2303088997 wait complete. 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 -- start start 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 Processor -- start 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61d2300 con 0x55cda60156b0 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61d2470 con 0x55cda61c3320 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda61d2630 con 0x55cda61c25c0 2023-11-17T00:03:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.400+0000 7f9af5e5ab80 10 client.0 _ll_register_callbacks cb 0x7f9ad4099d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50158/0 (socket says 172.21.15.23:50158) 2023-11-17T00:03:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 -- 172.21.15.23:0/817158793 learned_addr learned my addr 172.21.15.23:0/817158793 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 -- 172.21.15.23:0/817158793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 msgr2=0x55cda61ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 -- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 msgr2=0x55cda61d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.401+0000 7f9adbfff700 1 -- 172.21.15.23:0/817158793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad00278d0 con 0x55cda60156b0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9ae0b9c700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9adbfff700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40964b0 tx=0x7f9ad400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f562da1e86da7d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9abb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4099fb0 con 0x55cda60156b0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9acc01f040 con 0x55cda60156b0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f9ad40c9080 con 0x55cda60156b0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.402+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ad40a2400 con 0x55cda60156b0 2023-11-17T00:03:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 10 client.15507 fetch_fsmap learned FSMap version 24 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55cda5f21c00 con 0x55cda60156b0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9acc01f040 con 0x55cda60156b0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9ad4095da0 con 0x55cda60156b0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 10 client.15507 fetch_fsmap finished waiting for FSMap version 24 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 10 client.15507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 20 client.15507 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda622bca0 0x55cda622e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 4 client.15507 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.403+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cda60d9600 con 0x55cda622bca0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.404+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda622bca0 0x55cda622e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.404+0000 7f9adb7fe700 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda622bca0 0x55cda622e080 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87850faa437cdbb1 server_cookie=6e9f550b8296383 in_seq=0 out_seq=0 2023-11-17T00:03:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:26.404+0000 7f9abb7fe700 10 client.15507 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:26.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:26 smithi023 ceph-mon[97824]: pgmap v439: 129 pgs: 129 active+clean; 16 MiB data, 145 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 58 op/s 2023-11-17T00:03:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:26 smithi173 ceph-mon[131534]: pgmap v439: 129 pgs: 129 active+clean; 16 MiB data, 145 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 58 op/s 2023-11-17T00:03:26.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:26 smithi156 ceph-mon[131105]: pgmap v439: 129 pgs: 129 active+clean; 16 MiB data, 145 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 58 op/s 2023-11-17T00:03:28.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:28 smithi023 ceph-mon[97824]: pgmap v440: 129 pgs: 129 active+clean; 20 MiB data, 153 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 66 op/s 2023-11-17T00:03:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:28 smithi173 ceph-mon[131534]: pgmap v440: 129 pgs: 129 active+clean; 20 MiB data, 153 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 66 op/s 2023-11-17T00:03:28.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:28 smithi156 ceph-mon[131105]: pgmap v440: 129 pgs: 129 active+clean; 20 MiB data, 153 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 66 op/s 2023-11-17T00:03:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:29 smithi173 ceph-mon[131534]: pgmap v441: 129 pgs: 129 active+clean; 20 MiB data, 193 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 476 op/s 2023-11-17T00:03:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:29 smithi156 ceph-mon[131105]: pgmap v441: 129 pgs: 129 active+clean; 20 MiB data, 193 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 476 op/s 2023-11-17T00:03:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:29 smithi023 ceph-mon[97824]: pgmap v441: 129 pgs: 129 active+clean; 20 MiB data, 193 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 476 op/s 2023-11-17T00:03:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.705+0000 7f9abb7fe700 1 -- 172.21.15.23:0/817158793 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cda60d9600 con 0x55cda622bca0 2023-11-17T00:03:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.705+0000 7f9abb7fe700 10 client.15507 handle_command_reply: tid=0 2023-11-17T00:03:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.705+0000 7f9af5e5ab80 1 client.15507 shutdown 2023-11-17T00:03:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 msgr2=0x55cda61ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40964b0 tx=0x7f9ad400cf90 comp rx=0 tx=0).stop 2023-11-17T00:03:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 shutdown_connections 2023-11-17T00:03:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cda61c3320 0x55cda61d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55cda622bca0 0x55cda622e080 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda60156b0 0x55cda61ce860 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 --2- 172.21.15.23:0/817158793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cda61c25c0 0x55cda61ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 >> 172.21.15.23:0/817158793 conn(0x55cda61aef50 msgr2=0x55cda61c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.706+0000 7f9abb7fe700 0 client.15507 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.707+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 shutdown_connections 2023-11-17T00:03:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.707+0000 7f9af5e5ab80 1 -- 172.21.15.23:0/817158793 wait complete. 2023-11-17T00:03:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.707+0000 7f9af5e5ab80 20 client.15507 trim_cache size 0 max 16384 2023-11-17T00:03:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:03:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:03:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:03:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:03:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.708+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 msgr2=0x7f9ac40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.708+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 0x7f9ac40a3750 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0007c90 tx=0x7f9ad0007480 comp rx=0 tx=0).stop 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 msgr2=0x7f9adc1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400ef10 tx=0x7f9ad400c3a0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 shutdown_connections 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc0ffdb0 0x7f9adc19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9ac40a12a0 0x7f9ac40a3750 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9adc0feab0 0x7f9adc1981c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 --2- 172.21.15.23:0/1716289726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9adc0ff400 0x7f9adc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 >> 172.21.15.23:0/1716289726 conn(0x7f9adc0fa720 msgr2=0x7f9adc0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 shutdown_connections 2023-11-17T00:03:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:30.709+0000 7f9ae25ff700 1 -- 172.21.15.23:0/1716289726 wait complete. 2023-11-17T00:03:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:32 smithi023 ceph-mon[97824]: pgmap v442: 129 pgs: 129 active+clean; 25 MiB data, 218 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 770 op/s 2023-11-17T00:03:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:32 smithi173 ceph-mon[131534]: pgmap v442: 129 pgs: 129 active+clean; 25 MiB data, 218 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 770 op/s 2023-11-17T00:03:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:32 smithi156 ceph-mon[131105]: pgmap v442: 129 pgs: 129 active+clean; 25 MiB data, 218 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 770 op/s 2023-11-17T00:03:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:34 smithi023 ceph-mon[97824]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 271 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 1.73k op/s 2023-11-17T00:03:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:34 smithi173 ceph-mon[131534]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 271 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 1.73k op/s 2023-11-17T00:03:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:34 smithi156 ceph-mon[131105]: pgmap v443: 129 pgs: 129 active+clean; 25 MiB data, 271 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 1.73k op/s 2023-11-17T00:03:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:36 smithi023 ceph-mon[97824]: pgmap v444: 129 pgs: 129 active+clean; 8.8 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.20k op/s 2023-11-17T00:03:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:36 smithi173 ceph-mon[131534]: pgmap v444: 129 pgs: 129 active+clean; 8.8 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.20k op/s 2023-11-17T00:03:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:36 smithi156 ceph-mon[131105]: pgmap v444: 129 pgs: 129 active+clean; 8.8 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.20k op/s 2023-11-17T00:03:37.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:37 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:03:37.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:37 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:03:37.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:37 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:03:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:38 smithi023 ceph-mon[97824]: pgmap v445: 129 pgs: 129 active+clean; 13 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2023-11-17T00:03:38.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:38 smithi173 ceph-mon[131534]: pgmap v445: 129 pgs: 129 active+clean; 13 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2023-11-17T00:03:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:38 smithi156 ceph-mon[131105]: pgmap v445: 129 pgs: 129 active+clean; 13 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2023-11-17T00:03:40.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:39 smithi173 ceph-mon[131534]: pgmap v446: 129 pgs: 129 active+clean; 25 MiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 2.17k op/s 2023-11-17T00:03:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:39 smithi156 ceph-mon[131105]: pgmap v446: 129 pgs: 129 active+clean; 25 MiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 2.17k op/s 2023-11-17T00:03:40.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:39 smithi023 ceph-mon[97824]: pgmap v446: 129 pgs: 129 active+clean; 25 MiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 2.17k op/s 2023-11-17T00:03:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:42 smithi023 ceph-mon[97824]: pgmap v447: 129 pgs: 129 active+clean; 25 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.75k op/s 2023-11-17T00:03:42.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:42 smithi173 ceph-mon[131534]: pgmap v447: 129 pgs: 129 active+clean; 25 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.75k op/s 2023-11-17T00:03:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:42 smithi156 ceph-mon[131105]: pgmap v447: 129 pgs: 129 active+clean; 25 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 1.75k op/s 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.902+0000 7f4ca7fff700 1 -- 172.21.15.23:0/1437963427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca0105f00 msgr2=0x7f4ca0108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.902+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/1437963427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca0105f00 0x7f4ca0108360 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009f90 tx=0x7f4c9800b330 comp rx=0 tx=0).stop 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 -- 172.21.15.23:0/1437963427 shutdown_connections 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/1437963427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca0105f00 0x7f4ca0108360 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/1437963427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca01030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/1437963427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca01035e0 0x7f4ca01059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 -- 172.21.15.23:0/1437963427 >> 172.21.15.23:0/1437963427 conn(0x7f4ca00fa720 msgr2=0x7f4ca00fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 -- 172.21.15.23:0/1437963427 shutdown_connections 2023-11-17T00:03:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.903+0000 7f4ca7fff700 1 -- 172.21.15.23:0/1437963427 wait complete. 2023-11-17T00:03:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.905+0000 7f4ca7fff700 1 Processor -- start 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.905+0000 7f4ca7fff700 1 -- start start 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca01035e0 0x7f4ca010ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 0x7f4ca01090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0109870 con 0x7f4ca0105f00 2023-11-17T00:03:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01099e0 con 0x7f4ca0100cc0 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca7fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0109ba0 con 0x7f4ca01035e0 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:45798/0 (socket says 172.21.15.23:45798) 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/2687254952 learned_addr learned my addr 172.21.15.23:0/2687254952 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca659c700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 0x7f4ca01090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.906+0000 7f4ca559a700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca01035e0 0x7f4ca010ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/2687254952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca01035e0 msgr2=0x7f4ca010ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca01035e0 0x7f4ca010ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 msgr2=0x7f4ca01090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 0x7f4ca01090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/2687254952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98009c40 con 0x7f4ca0100cc0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca659c700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 0x7f4ca01090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.907+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e9c0 tx=0x7f4c9c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.908+0000 7f4c96ffd700 1 -- 172.21.15.23:0/2687254952 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c004a30 con 0x7f4ca0100cc0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.908+0000 7f4c96ffd700 1 -- 172.21.15.23:0/2687254952 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9c00cc80 con 0x7f4ca0100cc0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.908+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca010dea0 con 0x7f4ca0100cc0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.908+0000 7f4c96ffd700 1 -- 172.21.15.23:0/2687254952 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0054e0 con 0x7f4ca0100cc0 2023-11-17T00:03:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.908+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca010dd10 con 0x7f4ca0100cc0 2023-11-17T00:03:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.913+0000 7f4c96ffd700 1 -- 172.21.15.23:0/2687254952 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4c9c004b90 con 0x7f4ca0100cc0 2023-11-17T00:03:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.914+0000 7f4cbb9dfb80 1 Processor -- start 2023-11-17T00:03:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.914+0000 7f4c96ffd700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 0x7f4c8c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.914+0000 7f4c96ffd700 1 -- 172.21.15.23:0/2687254952 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4c9c014070 con 0x7f4ca0100cc0 2023-11-17T00:03:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 -- start start 2023-11-17T00:03:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4ca659c700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 0x7f4c8c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e46870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e47b10 0x5584b5e47f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5d4c000 con 0x5584b5ca26b0 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5d9e840 con 0x5584b5e47b10 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.915+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5d97cd0 con 0x5584b5e46db0 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39742/0 (socket says 172.21.15.23:39742) 2023-11-17T00:03:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 -- 172.21.15.23:0/990968210 learned_addr learned my addr 172.21.15.23:0/990968210 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e46870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 -- 172.21.15.23:0/990968210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e47b10 msgr2=0x5584b5e47f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e47b10 0x5584b5e47f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 -- 172.21.15.23:0/990968210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 msgr2=0x5584b5e46870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e46870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.916+0000 7f4ca559a700 1 -- 172.21.15.23:0/990968210 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b5c195d0 con 0x5584b5e46db0 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.917+0000 7f4ca559a700 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009af0 tx=0x7f4c90009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d635c053ecf8007f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.917+0000 7f4ca659c700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 0x7f4c8c0a36d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007900 tx=0x7f4c980075c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.917+0000 7f4c8bfff700 1 -- 172.21.15.23:0/990968210 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900095f0 con 0x5584b5e46db0 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.917+0000 7f4c8bfff700 1 -- 172.21.15.23:0/990968210 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9000ee50 con 0x5584b5e46db0 2023-11-17T00:03:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4c8bfff700 1 -- 172.21.15.23:0/990968210 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900123d0 con 0x5584b5e46db0 2023-11-17T00:03:43.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/990968210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 msgr2=0x5584b5e475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009af0 tx=0x7f4c90009e00 comp rx=0 tx=0).stop 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/990968210 shutdown_connections 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e47b10 0x5584b5e47f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e46870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.918+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/990968210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e475d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.919+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/990968210 >> 172.21.15.23:0/990968210 conn(0x5584b5e3bf50 msgr2=0x5584b5e41100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.919+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/990968210 shutdown_connections 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.919+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/990968210 wait complete. 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 -- start start 2023-11-17T00:03:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 Processor -- start 2023-11-17T00:03:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e4fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5d39880 con 0x5584b5ca26b0 2023-11-17T00:03:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5d39520 con 0x5584b5e46db0 2023-11-17T00:03:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.920+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5c1b8c0 con 0x5584b5e47b10 2023-11-17T00:03:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4cbb9dfb80 10 client.0 _ll_register_callbacks cb 0x5584b5d57720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:45810/0 (socket says 172.21.15.23:45810) 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 -- 172.21.15.23:0/195151946 learned_addr learned my addr 172.21.15.23:0/195151946 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca659c700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 -- 172.21.15.23:0/195151946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 msgr2=0x5584b5e572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 -- 172.21.15.23:0/195151946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 msgr2=0x5584b5e4fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e4fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.921+0000 7f4ca559a700 1 -- 172.21.15.23:0/195151946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c90009750 con 0x5584b5e46db0 2023-11-17T00:03:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4ca659c700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:03:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4ca559a700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4c90000c00 tx=0x7f4c9000ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc74a3eac9bcd83 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4c897fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:03:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9001c660 con 0x5584b5e46db0 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584b5e39710 con 0x5584b5e46db0 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.922+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4c9001c7c0 con 0x5584b5e46db0 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.923+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c9001c980 con 0x5584b5e46db0 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.923+0000 7f4cbb9dfb80 10 client.25532 fetch_fsmap learned FSMap version 25 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.923+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5584b5d5e130 con 0x5584b5e46db0 2023-11-17T00:03:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5584b5e39710 con 0x5584b5e46db0 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4c90009b20 con 0x5584b5e46db0 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 10 client.25532 fetch_fsmap finished waiting for FSMap version 25 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 10 client.25532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 20 client.25532 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5e5b9b0 0x5584b5e5ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 4 client.25532 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584b5d5a040 con 0x5584b5e5b9b0 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.924+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5e5b9b0 0x5584b5e5ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.925+0000 7f4c897fa700 10 client.25532 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.925+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5e5b9b0 0x5584b5e5ddc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5326d668e91c9c5 server_cookie=cf43c1721596a60b in_seq=0 out_seq=0 2023-11-17T00:03:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4c897fa700 1 -- 172.21.15.23:0/195151946 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5584b5d5a040 con 0x5584b5e5b9b0 2023-11-17T00:03:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4c897fa700 10 client.25532 handle_command_reply: tid=0 2023-11-17T00:03:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 client.25532 shutdown 2023-11-17T00:03:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 msgr2=0x5584b5e50270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4c90000c00 tx=0x7f4c9000ec70 comp rx=0 tx=0).stop 2023-11-17T00:03:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 shutdown_connections 2023-11-17T00:03:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5e5b9b0 0x5584b5e5ddc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5ca26b0 0x5584b5e4fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/195151946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5e46db0 0x5584b5e50270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 >> 172.21.15.23:0/195151946 conn(0x5584b5e3bf50 msgr2=0x5584b5e407e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.926+0000 7f4c897fa700 0 client.25532 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.927+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 shutdown_connections 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.928+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/195151946 wait complete. 2023-11-17T00:03:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.928+0000 7f4cbb9dfb80 20 client.25532 trim_cache size 0 max 16384 2023-11-17T00:03:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.952+0000 7f4cbb9dfb80 1 Processor -- start 2023-11-17T00:03:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.952+0000 7f4cbb9dfb80 1 -- start start 2023-11-17T00:03:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e4e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 0x5584b5e57270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e47420 con 0x5584b5e46db0 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e41c90 con 0x5584b5ca26b0 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e4dce0 con 0x5584b5e47b10 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4ca659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4ca659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39764/0 (socket says 172.21.15.23:39764) 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4ca659c700 1 -- 172.21.15.23:0/1075214554 learned_addr learned my addr 172.21.15.23:0/1075214554 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e4e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.953+0000 7f4ca559a700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 0x5584b5e57270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 -- 172.21.15.23:0/1075214554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 msgr2=0x5584b5e4e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e4e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 -- 172.21.15.23:0/1075214554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 msgr2=0x5584b5e57270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 0x5584b5e57270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 -- 172.21.15.23:0/1075214554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b5d5de80 con 0x5584b5e47b10 2023-11-17T00:03:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca559a700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 0x5584b5e57270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:03:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4ca659c700 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007ec0 tx=0x7f4c9801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd456481f8810709 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4c94ff9700 1 -- 172.21.15.23:0/1075214554 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800e040 con 0x5584b5e47b10 2023-11-17T00:03:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4c94ff9700 1 -- 172.21.15.23:0/1075214554 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9800be80 con 0x5584b5e47b10 2023-11-17T00:03:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.954+0000 7f4c94ff9700 1 -- 172.21.15.23:0/1075214554 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801c670 con 0x5584b5e47b10 2023-11-17T00:03:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/1075214554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 msgr2=0x5584b5e4df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007ec0 tx=0x7f4c9801e000 comp rx=0 tx=0).stop 2023-11-17T00:03:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/1075214554 shutdown_connections 2023-11-17T00:03:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e47b10 0x5584b5e4df30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e4e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/1075214554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e46db0 0x5584b5e57270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/1075214554 >> 172.21.15.23:0/1075214554 conn(0x5584b5e3bf50 msgr2=0x5584b5e626e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/1075214554 shutdown_connections 2023-11-17T00:03:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.955+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/1075214554 wait complete. 2023-11-17T00:03:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.956+0000 7f4cbb9dfb80 1 -- start start 2023-11-17T00:03:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.956+0000 7f4cbb9dfb80 1 Processor -- start 2023-11-17T00:03:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e53060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e47b10 0x5584b5e56080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e565c0 con 0x5584b5e47b10 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e56730 con 0x5584b5ca26b0 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b5e568a0 con 0x5584b5e46db0 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4cbb9dfb80 10 client.0 _ll_register_callbacks cb 0x7f4c90009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4ca5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:45834/0 (socket says 172.21.15.23:45834) 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.957+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/4279394419 learned_addr learned my addr 172.21.15.23:0/4279394419 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:03:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/4279394419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 msgr2=0x5584b5e53060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e53060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/4279394419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e47b10 msgr2=0x5584b5e56080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e47b10 0x5584b5e56080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 -- 172.21.15.23:0/4279394419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca00fee30 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4ca5d9b700 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005cb0 tx=0x7f4c9c099c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11986c9e15a4935f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4c897fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c098110 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.958+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4c9c005640 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.959+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5584b5d66600 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.959+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c9c099fd0 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.959+0000 7f4cbb9dfb80 10 client.25538 fetch_fsmap learned FSMap version 25 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.959+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5584b5baec00 con 0x5584b5ca26b0 2023-11-17T00:03:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.959+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5584b5d66600 con 0x5584b5ca26b0 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4c9c0a1d30 con 0x5584b5ca26b0 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 10 client.25538 fetch_fsmap finished waiting for FSMap version 25 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 20 client.25538 populate_metadata read hostname 'smithi023' 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5ec1330 0x5584b5ec3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 4 client.25538 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5584b5be6f10 con 0x5584b5ec1330 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.960+0000 7f4ca559a700 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5ec1330 0x5584b5ec3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.961+0000 7f4c897fa700 10 client.25538 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:43.961+0000 7f4ca559a700 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5ec1330 0x5584b5ec3740 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bceb711016ecef3f server_cookie=1e7cf4b2c159049 in_seq=0 out_seq=0 2023-11-17T00:03:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:44 smithi023 ceph-mon[97824]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 1.46k op/s 2023-11-17T00:03:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:44 smithi173 ceph-mon[131534]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 1.46k op/s 2023-11-17T00:03:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:44 smithi156 ceph-mon[131105]: pgmap v448: 129 pgs: 129 active+clean; 27 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 1.46k op/s 2023-11-17T00:03:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:46 smithi023 ceph-mon[97824]: pgmap v449: 129 pgs: 129 active+clean; 37 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 500 op/s 2023-11-17T00:03:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:46 smithi156 ceph-mon[131105]: pgmap v449: 129 pgs: 129 active+clean; 37 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 500 op/s 2023-11-17T00:03:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:46 smithi173 ceph-mon[131534]: pgmap v449: 129 pgs: 129 active+clean; 37 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 500 op/s 2023-11-17T00:03:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.925+0000 7f4c897fa700 1 -- 172.21.15.23:0/4279394419 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5584b5be6f10 con 0x5584b5ec1330 2023-11-17T00:03:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.925+0000 7f4c897fa700 10 client.25538 handle_command_reply: tid=0 2023-11-17T00:03:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.925+0000 7f4cbb9dfb80 1 client.25538 shutdown 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 msgr2=0x5584b5e52b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005cb0 tx=0x7f4c9c099c70 comp rx=0 tx=0).stop 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 shutdown_connections 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5584b5ec1330 0x5584b5ec3740 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584b5e47b10 0x5584b5e56080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584b5ca26b0 0x5584b5e52b20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 --2- 172.21.15.23:0/4279394419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584b5e46db0 0x5584b5e53060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4c897fa700 0 client.25538 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 >> 172.21.15.23:0/4279394419 conn(0x5584b5e3bf50 msgr2=0x5584b5e4df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 shutdown_connections 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 1 -- 172.21.15.23:0/4279394419 wait complete. 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.926+0000 7f4cbb9dfb80 20 client.25538 trim_cache size 0 max 16384 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:03:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 msgr2=0x7f4c8c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 0x7f4c8c0a36d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007900 tx=0x7f4c980075c0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 msgr2=0x7f4ca010c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e9c0 tx=0x7f4c9c00ecd0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 shutdown_connections 2023-11-17T00:03:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ca0105f00 0x7f4ca01090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ca0100cc0 0x7f4ca010c540 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f4c8c0a1220 0x7f4c8c0a36d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 --2- 172.21.15.23:0/2687254952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ca01035e0 0x7f4ca010ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:03:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 >> 172.21.15.23:0/2687254952 conn(0x7f4ca00fa720 msgr2=0x7f4ca0101920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:03:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 shutdown_connections 2023-11-17T00:03:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:03:47.927+0000 7f4ca7fff700 1 -- 172.21.15.23:0/2687254952 wait complete. 2023-11-17T00:03:48.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:48 smithi023 ceph-mon[97824]: pgmap v450: 129 pgs: 129 active+clean; 37 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2023-11-17T00:03:48.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:48 smithi173 ceph-mon[131534]: pgmap v450: 129 pgs: 129 active+clean; 37 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2023-11-17T00:03:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:48 smithi156 ceph-mon[131105]: pgmap v450: 129 pgs: 129 active+clean; 37 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2023-11-17T00:03:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:49 smithi173 ceph-mon[131534]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 987 op/s 2023-11-17T00:03:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:49 smithi156 ceph-mon[131105]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 987 op/s 2023-11-17T00:03:50.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:49 smithi023 ceph-mon[97824]: pgmap v451: 129 pgs: 129 active+clean; 36 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 987 op/s 2023-11-17T00:03:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:52 smithi023 ceph-mon[97824]: pgmap v452: 129 pgs: 129 active+clean; 37 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 1.15k op/s 2023-11-17T00:03:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:52 smithi173 ceph-mon[131534]: pgmap v452: 129 pgs: 129 active+clean; 37 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 1.15k op/s 2023-11-17T00:03:52.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:52 smithi156 ceph-mon[131105]: pgmap v452: 129 pgs: 129 active+clean; 37 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 1.15k op/s 2023-11-17T00:03:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:54 smithi023 ceph-mon[97824]: pgmap v453: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.29k op/s 2023-11-17T00:04:04.943 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:56 smithi023 ceph-mon[97824]: pgmap v454: 129 pgs: 129 active+clean; 36 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 1.30k op/s 2023-11-17T00:04:05.320 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:58 smithi023 ceph-mon[97824]: pgmap v455: 129 pgs: 129 active+clean; 40 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2023-11-17T00:04:05.320 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:03:59 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:05.321 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:00 smithi023 ceph-mon[97824]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 1.06k op/s 2023-11-17T00:04:05.929 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:54 smithi156 ceph-mon[131105]: pgmap v453: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.29k op/s 2023-11-17T00:04:05.929 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:56 smithi156 ceph-mon[131105]: pgmap v454: 129 pgs: 129 active+clean; 36 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 1.30k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:58 smithi156 ceph-mon[131105]: pgmap v455: 129 pgs: 129 active+clean; 40 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:03:59 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:00 smithi156 ceph-mon[131105]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 1.06k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:02 smithi156 ceph-mon[131105]: pgmap v457: 129 pgs: 129 active+clean; 53 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:04 smithi156 ceph-mon[131105]: pgmap v458: 129 pgs: 129 active+clean; 61 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 572 op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:54 smithi173 ceph-mon[131534]: pgmap v453: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.29k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:56 smithi173 ceph-mon[131534]: pgmap v454: 129 pgs: 129 active+clean; 36 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 1.30k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:58 smithi173 ceph-mon[131534]: pgmap v455: 129 pgs: 129 active+clean; 40 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:03:59 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:00 smithi173 ceph-mon[131534]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 1.06k op/s 2023-11-17T00:04:05.930 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:02 smithi173 ceph-mon[131534]: pgmap v457: 129 pgs: 129 active+clean; 53 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2023-11-17T00:04:05.931 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:04 smithi173 ceph-mon[131534]: pgmap v458: 129 pgs: 129 active+clean; 61 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 572 op/s 2023-11-17T00:04:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 -- 172.21.15.23:0/4288791831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c100240 msgr2=0x7ff26c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 --2- 172.21.15.23:0/4288791831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c100240 0x7ff26c100650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009990 tx=0x7ff25c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:04:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 -- 172.21.15.23:0/4288791831 shutdown_connections 2023-11-17T00:04:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 --2- 172.21.15.23:0/4288791831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c101dc0 0x7ff26c108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 --2- 172.21.15.23:0/4288791831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c100240 0x7ff26c100650 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 --2- 172.21.15.23:0/4288791831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101430 0x7ff26c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 -- 172.21.15.23:0/4288791831 >> 172.21.15.23:0/4288791831 conn(0x7ff26c0fb840 msgr2=0x7ff26c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 -- 172.21.15.23:0/4288791831 shutdown_connections 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.097+0000 7ff270fff700 1 -- 172.21.15.23:0/4288791831 wait complete. 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.098+0000 7ff270fff700 1 Processor -- start 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 -- start start 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c100240 0x7ff26c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:06.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101dc0 0x7ff26c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:06.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c19fda0 con 0x7ff26c100240 2023-11-17T00:04:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c19fee0 con 0x7ff26c101430 2023-11-17T00:04:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff270fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c1a0070 con 0x7ff26c101dc0 2023-11-17T00:04:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:36388/0 (socket says 172.21.15.23:36388) 2023-11-17T00:04:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff26ad9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101dc0 0x7ff26c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff269d9b700 1 -- 172.21.15.23:0/478711961 learned_addr learned my addr 172.21.15.23:0/478711961 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c100240 0x7ff26c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.099+0000 7ff269d9b700 1 -- 172.21.15.23:0/478711961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101dc0 msgr2=0x7ff26c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff269d9b700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101dc0 0x7ff26c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:06.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff269d9b700 1 -- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c100240 msgr2=0x7ff26c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff269d9b700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c100240 0x7ff26c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff269d9b700 1 -- 172.21.15.23:0/478711961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0095f0 con 0x7ff26c101430 2023-11-17T00:04:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff269d9b700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e9a0 tx=0x7ff26000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff25b7fe700 1 -- 172.21.15.23:0/478711961 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260009790 con 0x7ff26c101430 2023-11-17T00:04:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff25b7fe700 1 -- 172.21.15.23:0/478711961 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff260010430 con 0x7ff26c101430 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c10bbf0 con 0x7ff26c101430 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff25b7fe700 1 -- 172.21.15.23:0/478711961 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260018680 con 0x7ff26c101430 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.100+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c10ba60 con 0x7ff26c101430 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.104+0000 7ff25b7fe700 1 -- 172.21.15.23:0/478711961 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff2600105a0 con 0x7ff26c101430 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 Processor -- start 2023-11-17T00:04:07.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 -- start start 2023-11-17T00:04:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff25b7fe700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 0x7ff2540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a358870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff25b7fe700 1 -- 172.21.15.23:0/478711961 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7ff260014070 con 0x7ff26c101430 2023-11-17T00:04:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a359f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a25e000 con 0x56358a359b10 2023-11-17T00:04:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a2b0840 con 0x56358a1b46b0 2023-11-17T00:04:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff284857b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a2a9cd0 con 0x56358a358db0 2023-11-17T00:04:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:54684/0 (socket says 172.21.15.23:54684) 2023-11-17T00:04:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.105+0000 7ff269d9b700 1 -- 172.21.15.23:0/2873960143 learned_addr learned my addr 172.21.15.23:0/2873960143 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff26a59c700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a358870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 0x7ff2540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 -- 172.21.15.23:0/2873960143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 msgr2=0x56358a358870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a358870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a359f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 -- 172.21.15.23:0/2873960143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 msgr2=0x56358a359f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a359f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 -- 172.21.15.23:0/2873960143 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358a12b5d0 con 0x56358a358db0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a359f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff269d9b700 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff260099e70 tx=0x7ff260099940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80b6d852458ecd46 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff258ff9700 1 -- 172.21.15.23:0/2873960143 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600a2330 con 0x56358a358db0 2023-11-17T00:04:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 0x7ff2540a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff264009ec0 tx=0x7ff264009920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff258ff9700 1 -- 172.21.15.23:0/2873960143 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff260098280 con 0x56358a358db0 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.106+0000 7ff258ff9700 1 -- 172.21.15.23:0/2873960143 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600a2100 con 0x56358a358db0 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 -- 172.21.15.23:0/2873960143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 msgr2=0x56358a3595d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff260099e70 tx=0x7ff260099940 comp rx=0 tx=0).stop 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 -- 172.21.15.23:0/2873960143 shutdown_connections 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a359f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a358870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 --2- 172.21.15.23:0/2873960143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a3595d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.107+0000 7ff284857b80 1 -- 172.21.15.23:0/2873960143 >> 172.21.15.23:0/2873960143 conn(0x56358a34df50 msgr2=0x56358a353100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.108+0000 7ff284857b80 1 -- 172.21.15.23:0/2873960143 shutdown_connections 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.108+0000 7ff284857b80 1 -- 172.21.15.23:0/2873960143 wait complete. 2023-11-17T00:04:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.108+0000 7ff284857b80 1 -- start start 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 Processor -- start 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a358db0 0x56358a36aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a359b10 0x56358a371a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a24b880 con 0x56358a358db0 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a24b520 con 0x56358a1b46b0 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a12d8c0 con 0x56358a359b10 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff284857b80 10 client.0 _ll_register_callbacks cb 0x56358a269720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.109+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:36400/0 (socket says 172.21.15.23:36400) 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 -- 172.21.15.23:0/3206448510 learned_addr learned my addr 172.21.15.23:0/3206448510 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff269d9b700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a358db0 0x56358a36aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a359b10 0x56358a371a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 -- 172.21.15.23:0/3206448510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a359b10 msgr2=0x56358a371a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a359b10 0x56358a371a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 -- 172.21.15.23:0/3206448510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a358db0 msgr2=0x56358a36aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a358db0 0x56358a36aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 -- 172.21.15.23:0/3206448510 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c10ba60 con 0x56358a1b46b0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff26a59c700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff26c101290 tx=0x7ff25c005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b4c84900bf66c77 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.110+0000 7ff251ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c01d030 con 0x56358a1b46b0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56358a34b710 con 0x56358a1b46b0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7ff25c004500 con 0x56358a1b46b0 2023-11-17T00:04:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff25c0046c0 con 0x56358a1b46b0 2023-11-17T00:04:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff284857b80 10 client.25550 fetch_fsmap learned FSMap version 26 2023-11-17T00:04:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.111+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56358a270130 con 0x56358a1b46b0 2023-11-17T00:04:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56358a34b710 con 0x56358a1b46b0 2023-11-17T00:04:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ff25c00f710 con 0x56358a1b46b0 2023-11-17T00:04:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 10 client.25550 fetch_fsmap finished waiting for FSMap version 26 2023-11-17T00:04:07.776 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 10 client.25550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 20 client.25550 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3762f0 0x56358a378700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 4 client.25550 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56358a26c040 con 0x56358a3762f0 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.112+0000 7ff269d9b700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3762f0 0x56358a378700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.113+0000 7ff269d9b700 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3762f0 0x56358a378700 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e250d8f6cb2659 server_cookie=5d2e88723f476ec7 in_seq=0 out_seq=0 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.113+0000 7ff251ffb700 10 client.25550 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.114+0000 7ff251ffb700 1 -- 172.21.15.23:0/3206448510 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56358a26c040 con 0x56358a3762f0 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.114+0000 7ff251ffb700 10 client.25550 handle_command_reply: tid=0 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.114+0000 7ff284857b80 1 client.25550 shutdown 2023-11-17T00:04:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.114+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 msgr2=0x56358a36a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.114+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff26c101290 tx=0x7ff25c005750 comp rx=0 tx=0).stop 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 shutdown_connections 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a359b10 0x56358a371a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36a4e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3762f0 0x56358a378700 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 --2- 172.21.15.23:0/3206448510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a358db0 0x56358a36aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 >> 172.21.15.23:0/3206448510 conn(0x56358a34df50 msgr2=0x56358a3527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff251ffb700 0 client.25550 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 shutdown_connections 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 1 -- 172.21.15.23:0/3206448510 wait complete. 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.115+0000 7ff284857b80 20 client.25550 trim_cache size 0 max 16384 2023-11-17T00:04:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.131+0000 7ff284857b80 1 Processor -- start 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 -- start start 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 0x56358a371a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a35cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a34dbc0 con 0x56358a359b10 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a3611b0 con 0x56358a358db0 2023-11-17T00:04:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff284857b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a353c90 con 0x56358a1b46b0 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 0x56358a371a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a35cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:54692/0 (socket says 172.21.15.23:54692) 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 0x56358a371a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:36406/0 (socket says 172.21.15.23:36406) 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 -- 172.21.15.23:0/357028592 learned_addr learned my addr 172.21.15.23:0/357028592 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 -- 172.21.15.23:0/357028592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 msgr2=0x56358a371a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 0x56358a371a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 -- 172.21.15.23:0/357028592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 msgr2=0x56358a35cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.132+0000 7ff26a59c700 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a35cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.133+0000 7ff26a59c700 1 -- 172.21.15.23:0/357028592 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358a26fe80 con 0x56358a1b46b0 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.133+0000 7ff26a59c700 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc233c3caa0612ee server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.133+0000 7ff2597fa700 1 -- 172.21.15.23:0/357028592 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c01d040 con 0x56358a1b46b0 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.133+0000 7ff2597fa700 1 -- 172.21.15.23:0/357028592 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff25c01e040 con 0x56358a1b46b0 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.133+0000 7ff2597fa700 1 -- 172.21.15.23:0/357028592 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c004b60 con 0x56358a1b46b0 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- 172.21.15.23:0/357028592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 msgr2=0x56358a35c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c005f00 comp rx=0 tx=0).stop 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- 172.21.15.23:0/357028592 shutdown_connections 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a35cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a1b46b0 0x56358a35c9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 --2- 172.21.15.23:0/357028592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a358db0 0x56358a371a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- 172.21.15.23:0/357028592 >> 172.21.15.23:0/357028592 conn(0x56358a34df50 msgr2=0x56358a3609d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- 172.21.15.23:0/357028592 shutdown_connections 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- 172.21.15.23:0/357028592 wait complete. 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 -- start start 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.134+0000 7ff284857b80 1 Processor -- start 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a370d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a371240 con 0x56358a359b10 2023-11-17T00:04:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a3713b0 con 0x56358a1b46b0 2023-11-17T00:04:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358a371570 con 0x56358a358db0 2023-11-17T00:04:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff284857b80 10 client.0 _ll_register_callbacks cb 0x7ff25c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff26a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:36420/0 (socket says 172.21.15.23:36420) 2023-11-17T00:04:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff26a59c700 1 -- 172.21.15.23:0/1005301251 learned_addr learned my addr 172.21.15.23:0/1005301251 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff269d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:54706/0 (socket says 172.21.15.23:54706) 2023-11-17T00:04:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.135+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a370d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 -- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 msgr2=0x56358a36dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 -- 172.21.15.23:0/1005301251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 msgr2=0x56358a370d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a370d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 -- 172.21.15.23:0/1005301251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56358a270130 con 0x56358a358db0 2023-11-17T00:04:08.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff26ad9d700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a370d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:04:08.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff26a59c700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:08.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff269d9b700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff260096950 tx=0x7ff260098470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8df769a42495d292 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff251ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:04:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260010880 con 0x56358a358db0 2023-11-17T00:04:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff260098280 con 0x56358a358db0 2023-11-17T00:04:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7ff2600b81c0 con 0x56358a358db0 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2600109e0 con 0x56358a358db0 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff284857b80 10 client.25591 fetch_fsmap learned FSMap version 26 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.136+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56358a0c0c00 con 0x56358a358db0 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff260064020 con 0x56358a358db0 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 26) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ff2600a1ea0 con 0x56358a358db0 2023-11-17T00:04:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 10 client.25591 fetch_fsmap finished waiting for FSMap version 26 2023-11-17T00:04:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 20 client.25591 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3cae60 0x56358a3cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 4 client.25591 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:04:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56358a278600 con 0x56358a3cae60 2023-11-17T00:04:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.137+0000 7ff26a59c700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3cae60 0x56358a3cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.138+0000 7ff26a59c700 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3cae60 0x56358a3cd270 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=108fb031d0b2d718 server_cookie=c1d00d6d7d064172 in_seq=0 out_seq=0 2023-11-17T00:04:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:01.138+0000 7ff251ffb700 10 client.25591 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:08.718 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:02 smithi023 ceph-mon[97824]: pgmap v457: 129 pgs: 129 active+clean; 53 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2023-11-17T00:04:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff251ffb700 1 -- 172.21.15.23:0/1005301251 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56358a278600 con 0x56358a3cae60 2023-11-17T00:04:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff251ffb700 10 client.25591 handle_command_reply: tid=0 2023-11-17T00:04:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 client.25591 shutdown 2023-11-17T00:04:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 msgr2=0x56358a36e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff260096950 tx=0x7ff260098470 comp rx=0 tx=0).stop 2023-11-17T00:04:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 shutdown_connections 2023-11-17T00:04:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56358a3cae60 0x56358a3cd270 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56358a359b10 0x56358a370d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358a1b46b0 0x56358a36dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 --2- 172.21.15.23:0/1005301251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56358a358db0 0x56358a36e190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 >> 172.21.15.23:0/1005301251 conn(0x56358a34df50 msgr2=0x56358a35fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.990+0000 7ff251ffb700 0 client.25591 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.991+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 shutdown_connections 2023-11-17T00:04:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.991+0000 7ff284857b80 1 -- 172.21.15.23:0/1005301251 wait complete. 2023-11-17T00:04:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.991+0000 7ff284857b80 20 client.25591 trim_cache size 0 max 16384 2023-11-17T00:04:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:04:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:04:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:04:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:04:08.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:06 smithi156 ceph-mon[131105]: pgmap v459: 129 pgs: 129 active+clean; 64 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.30k op/s 2023-11-17T00:04:08.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:08 smithi156 ceph-mon[131105]: pgmap v460: 129 pgs: 129 active+clean; 65 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.73k op/s 2023-11-17T00:04:08.728 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:06 smithi173 ceph-mon[131534]: pgmap v459: 129 pgs: 129 active+clean; 64 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.30k op/s 2023-11-17T00:04:08.729 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:08 smithi173 ceph-mon[131534]: pgmap v460: 129 pgs: 129 active+clean; 65 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.73k op/s 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 msgr2=0x7ff2540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 0x7ff2540a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff264009ec0 tx=0x7ff264009920 comp rx=0 tx=0).stop 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 msgr2=0x7ff26c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e9a0 tx=0x7ff26000ecb0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 shutdown_connections 2023-11-17T00:04:08.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2540a1150 0x7ff2540a3600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff26c101dc0 0x7ff26c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff26c100240 0x7ff26c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 --2- 172.21.15.23:0/478711961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26c101430 0x7ff26c198760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:08.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 >> 172.21.15.23:0/478711961 conn(0x7ff26c0fb840 msgr2=0x7ff26c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:08.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 shutdown_connections 2023-11-17T00:04:08.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:03.992+0000 7ff270fff700 1 -- 172.21.15.23:0/478711961 wait complete. 2023-11-17T00:04:08.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:04 smithi023 ceph-mon[97824]: pgmap v458: 129 pgs: 129 active+clean; 61 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 572 op/s 2023-11-17T00:04:08.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:06 smithi023 ceph-mon[97824]: pgmap v459: 129 pgs: 129 active+clean; 64 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.30k op/s 2023-11-17T00:04:08.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:08 smithi023 ceph-mon[97824]: pgmap v460: 129 pgs: 129 active+clean; 65 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.73k op/s 2023-11-17T00:04:10.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:09 smithi023 ceph-mon[97824]: pgmap v461: 129 pgs: 129 active+clean; 58 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 1.89k op/s 2023-11-17T00:04:10.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:09 smithi173 ceph-mon[131534]: pgmap v461: 129 pgs: 129 active+clean; 58 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 1.89k op/s 2023-11-17T00:04:10.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:09 smithi156 ceph-mon[131105]: pgmap v461: 129 pgs: 129 active+clean; 58 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 1.89k op/s 2023-11-17T00:04:10.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[191646]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:10.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[191646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:10.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[191646]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:10.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[191646]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:11.268 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[192845]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:11.268 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[192845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:11.268 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[192845]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:11.268 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:04:10 smithi023 sudo[192845]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:11.513 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158727]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:11.513 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158727]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:11.513 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158727]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:11.513 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158727]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:11.857 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158732]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:11.857 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:11.857 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158732]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:11.857 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:04:11 smithi156 sudo[158732]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:12.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:12.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:11 smithi173 ceph-mon[131534]: pgmap v462: 129 pgs: 129 active+clean; 58 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 1.79k op/s 2023-11-17T00:04:12.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:11 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:12.105 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:11 smithi173 sudo[160021]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:12.105 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:11 smithi173 sudo[160021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:12.105 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:11 smithi173 sudo[160021]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:12.105 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:11 smithi173 sudo[160021]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:12.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:12.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:11 smithi023 ceph-mon[97824]: pgmap v462: 129 pgs: 129 active+clean; 58 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 1.79k op/s 2023-11-17T00:04:12.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:11 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:12.357 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:12 smithi173 sudo[160024]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:12.357 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:12 smithi173 sudo[160024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:12.357 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:12 smithi173 sudo[160024]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:12.357 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:04:12 smithi173 sudo[160024]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:11 smithi156 ceph-mon[131105]: pgmap v462: 129 pgs: 129 active+clean; 58 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.8 MiB/s wr, 1.79k op/s 2023-11-17T00:04:12.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:11 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:13.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:13 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:13.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:13 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:13.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:13 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:14.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:13 smithi023 sudo[193515]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:14.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:13 smithi023 sudo[193515]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:14.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:13 smithi023 sudo[193515]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:14.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:13 smithi023 sudo[193515]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:14.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:14 smithi023 ceph-mon[97824]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 1.76k op/s 2023-11-17T00:04:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:14 smithi173 ceph-mon[131534]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 1.76k op/s 2023-11-17T00:04:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:14 smithi156 ceph-mon[131105]: pgmap v463: 129 pgs: 129 active+clean; 69 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 1.76k op/s 2023-11-17T00:04:15.018 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:14 smithi023 sudo[194830]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:15.018 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:14 smithi023 sudo[194830]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:15.018 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:14 smithi023 sudo[194830]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:15.018 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:04:14 smithi023 sudo[194830]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:15.211 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:14 smithi156 sudo[158740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:15.211 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:14 smithi156 sudo[158740]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:15.211 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:14 smithi156 sudo[158740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:15.211 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:14 smithi156 sudo[158740]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:15.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:15 smithi156 sudo[158743]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:15.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:15 smithi156 sudo[158743]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:15.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:15 smithi156 sudo[158743]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:15.607 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:04:15 smithi156 sudo[158743]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:15.803 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160030]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:15.803 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160030]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:15.803 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160030]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:15.803 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160030]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:16.107 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160033]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:16.107 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160033]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:16.108 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160033]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:16.108 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:04:15 smithi173 sudo[160033]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:16.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:16 smithi023 ceph-mon[97824]: pgmap v464: 129 pgs: 129 active+clean; 81 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 1.57k op/s 2023-11-17T00:04:16.467 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:16 smithi023 sudo[196391]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:16.467 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:16 smithi023 sudo[196391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:16.467 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:16 smithi023 sudo[196391]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:16.468 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:16 smithi023 sudo[196391]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:16 smithi173 ceph-mon[131534]: pgmap v464: 129 pgs: 129 active+clean; 81 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 1.57k op/s 2023-11-17T00:04:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:16 smithi156 ceph-mon[131105]: pgmap v464: 129 pgs: 129 active+clean; 81 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 1.57k op/s 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 -- 172.21.15.23:0/1008088632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64100240 msgr2=0x7f8b64100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 --2- 172.21.15.23:0/1008088632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64100240 0x7f8b64100650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009990 tx=0x7f8b54009ca0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 -- 172.21.15.23:0/1008088632 shutdown_connections 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 --2- 172.21.15.23:0/1008088632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101dc0 0x7f8b64108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 --2- 172.21.15.23:0/1008088632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64100240 0x7f8b64100650 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 --2- 172.21.15.23:0/1008088632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64101430 0x7f8b64101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 -- 172.21.15.23:0/1008088632 >> 172.21.15.23:0/1008088632 conn(0x7f8b640fb840 msgr2=0x7f8b640fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.153+0000 7f8b699ff700 1 -- 172.21.15.23:0/1008088632 shutdown_connections 2023-11-17T00:04:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.154+0000 7f8b699ff700 1 -- 172.21.15.23:0/1008088632 wait complete. 2023-11-17T00:04:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.154+0000 7f8b699ff700 1 Processor -- start 2023-11-17T00:04:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 -- start start 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101430 0x7f8b64072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64101dc0 0x7f8b64073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64073bc0 con 0x7f8b64101430 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64073d30 con 0x7f8b64101dc0 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b699ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b641a0090 con 0x7f8b64100240 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b627fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101430 0x7f8b64072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:40796/0 (socket says 172.21.15.23:40796) 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b62ffd700 1 -- 172.21.15.23:0/2731280822 learned_addr learned my addr 172.21.15.23:0/2731280822 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.155+0000 7f8b637fe700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64101dc0 0x7f8b64073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 -- 172.21.15.23:0/2731280822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64101dc0 msgr2=0x7f8b64073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64101dc0 0x7f8b64073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 -- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101430 msgr2=0x7f8b64072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101430 0x7f8b64072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 -- 172.21.15.23:0/2731280822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540095f0 con 0x7f8b64100240 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8b54004730 tx=0x7f8b54004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b689fd700 1 -- 172.21.15.23:0/2731280822 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5401d070 con 0x7f8b64100240 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b689fd700 1 -- 172.21.15.23:0/2731280822 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b54021410 con 0x7f8b64100240 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6410bbf0 con 0x7f8b64100240 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b689fd700 1 -- 172.21.15.23:0/2731280822 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54017650 con 0x7f8b64100240 2023-11-17T00:04:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.156+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6410ba60 con 0x7f8b64100240 2023-11-17T00:04:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.160+0000 7f8b689fd700 1 -- 172.21.15.23:0/2731280822 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b54021580 con 0x7f8b64100240 2023-11-17T00:04:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 Processor -- start 2023-11-17T00:04:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 -- start start 2023-11-17T00:04:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b689fd700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 0x7f8b4c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b689fd700 1 -- 172.21.15.23:0/2731280822 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f8b540ce040 con 0x7f8b64100240 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 0x5615059035d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505808000 con 0x56150575e6b0 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150585a840 con 0x561505903b10 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.161+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505853cd0 con 0x561505902db0 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b637fe700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 0x7f8b4c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35206/0 (socket says 172.21.15.23:35206) 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b62ffd700 1 -- 172.21.15.23:0/3643902539 learned_addr learned my addr 172.21.15.23:0/3643902539 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 0x5615059035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b637fe700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 -- 172.21.15.23:0/3643902539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 msgr2=0x561505903f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 -- 172.21.15.23:0/3643902539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 msgr2=0x561505902870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 -- 172.21.15.23:0/3643902539 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615056d55d0 con 0x561505902db0 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b637fe700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 0x7f8b4c0a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009ec0 tx=0x7f8b5c0134a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.162+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 0x5615059035d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009cb0 tx=0x7f8b58011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db11cfcb7093ed6 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b49ffb700 1 -- 172.21.15.23:0/3643902539 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58011d20 con 0x561505902db0 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b49ffb700 1 -- 172.21.15.23:0/3643902539 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b58011e80 con 0x561505902db0 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b49ffb700 1 -- 172.21.15.23:0/3643902539 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b580193e0 con 0x561505902db0 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3643902539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 msgr2=0x5615059035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 0x5615059035d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009cb0 tx=0x7f8b58011440 comp rx=0 tx=0).stop 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3643902539 shutdown_connections 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x561505902870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3643902539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505902db0 0x5615059035d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3643902539 >> 172.21.15.23:0/3643902539 conn(0x5615058f7f50 msgr2=0x5615058fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3643902539 shutdown_connections 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.163+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3643902539 wait complete. 2023-11-17T00:04:17.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.164+0000 7f8b7d281b80 1 -- start start 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.164+0000 7f8b7d281b80 1 Processor -- start 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505902db0 0x56150590c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505913190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615057f5880 con 0x56150575e6b0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615057f5520 con 0x561505902db0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615056d78c0 con 0x561505903b10 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b7d281b80 10 client.0 _ll_register_callbacks cb 0x561505813720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b637fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505913190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b627fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505902db0 0x56150590c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35218/0 (socket says 172.21.15.23:35218) 2023-11-17T00:04:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b637fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505913190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:40816/0 (socket says 172.21.15.23:40816) 2023-11-17T00:04:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 -- 172.21.15.23:0/3734812031 learned_addr learned my addr 172.21.15.23:0/3734812031 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 -- 172.21.15.23:0/3734812031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 msgr2=0x561505913190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505913190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 -- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505902db0 msgr2=0x56150590c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505902db0 0x56150590c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 -- 172.21.15.23:0/3734812031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b580098e0 con 0x56150575e6b0 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8b540098e0 tx=0x7f8b54021b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7a5ca72ccc5616 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.165+0000 7f8b42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:04:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540b2070 con 0x56150575e6b0 2023-11-17T00:04:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f8b5409b190 con 0x56150575e6b0 2023-11-17T00:04:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561505816040 con 0x56150575e6b0 2023-11-17T00:04:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b5409f1c0 con 0x56150575e6b0 2023-11-17T00:04:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 10 client.15543 fetch_fsmap learned FSMap version 26 2023-11-17T00:04:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56150581a130 con 0x56150575e6b0 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561505816040 con 0x56150575e6b0 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8b540a30c0 con 0x56150575e6b0 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 10 client.15543 fetch_fsmap finished waiting for FSMap version 26 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 20 client.15543 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5615059176d0 0x561505919ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 4 client.15543 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.166+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561505822600 con 0x5615059176d0 2023-11-17T00:04:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.167+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5615059176d0 0x561505919ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.167+0000 7f8b627fc700 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5615059176d0 0x561505919ae0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8d8ffa079c5005 server_cookie=71affb4d6a7909c3 in_seq=0 out_seq=0 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.167+0000 7f8b42ffd700 10 client.15543 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b42ffd700 1 -- 172.21.15.23:0/3734812031 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561505822600 con 0x5615059176d0 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b42ffd700 10 client.15543 handle_command_reply: tid=0 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 client.15543 shutdown 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 msgr2=0x56150590bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8b540098e0 tx=0x7f8b54021b40 comp rx=0 tx=0).stop 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 shutdown_connections 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505913190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5615059176d0 0x561505919ae0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56150575e6b0 0x56150590bc10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/3734812031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505902db0 0x56150590c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b42ffd700 0 client.15543 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 >> 172.21.15.23:0/3734812031 conn(0x5615058f7f50 msgr2=0x5615058fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 shutdown_connections 2023-11-17T00:04:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.169+0000 7f8b7d281b80 1 -- 172.21.15.23:0/3734812031 wait complete. 2023-11-17T00:04:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.170+0000 7f8b7d281b80 20 client.15543 trim_cache size 0 max 16384 2023-11-17T00:04:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.185+0000 7f8b7d281b80 1 Processor -- start 2023-11-17T00:04:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.185+0000 7f8b7d281b80 1 -- start start 2023-11-17T00:04:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x561505913150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150590b0c0 con 0x561505902db0 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505903420 con 0x561505903b10 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505904420 con 0x56150575e6b0 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b627fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x561505913150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:40818/0 (socket says 172.21.15.23:40818) 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b637fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 -- 172.21.15.23:0/346162125 learned_addr learned my addr 172.21.15.23:0/346162125 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 -- 172.21.15.23:0/346162125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 msgr2=0x561505903ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 -- 172.21.15.23:0/346162125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 msgr2=0x561505913150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x561505913150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b62ffd700 1 -- 172.21.15.23:0/346162125 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561505819e80 con 0x56150575e6b0 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.186+0000 7f8b627fc700 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x561505913150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8b54097c00 tx=0x7f8b54099740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf06b843a85571e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b4a7fc700 1 -- 172.21.15.23:0/346162125 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540b3b30 con 0x56150575e6b0 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b4a7fc700 1 -- 172.21.15.23:0/346162125 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b5409b190 con 0x56150575e6b0 2023-11-17T00:04:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b4a7fc700 1 -- 172.21.15.23:0/346162125 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540a30c0 con 0x56150575e6b0 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 -- 172.21.15.23:0/346162125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 msgr2=0x5615058f9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8b54097c00 tx=0x7f8b54099740 comp rx=0 tx=0).stop 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 -- 172.21.15.23:0/346162125 shutdown_connections 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561505903b10 0x561505903ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56150575e6b0 0x5615058f9420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/346162125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x561505913150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.187+0000 7f8b7d281b80 1 -- 172.21.15.23:0/346162125 >> 172.21.15.23:0/346162125 conn(0x5615058f7f50 msgr2=0x5615058fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.188+0000 7f8b7d281b80 1 -- 172.21.15.23:0/346162125 shutdown_connections 2023-11-17T00:04:17.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.188+0000 7f8b7d281b80 1 -- 172.21.15.23:0/346162125 wait complete. 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.188+0000 7f8b7d281b80 1 -- start start 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.188+0000 7f8b7d281b80 1 Processor -- start 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x56150590ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505911e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505912390 con 0x561505902db0 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561505912500 con 0x56150575e6b0 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615059126c0 con 0x561505903b10 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b7d281b80 10 client.0 _ll_register_callbacks cb 0x561505813720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:46544/0 (socket says 172.21.15.23:46544) 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 -- 172.21.15.23:0/1509906735 learned_addr learned my addr 172.21.15.23:0/1509906735 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b637fe700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505911e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b627fc700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x56150590ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 -- 172.21.15.23:0/1509906735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 msgr2=0x561505911e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505911e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 -- 172.21.15.23:0/1509906735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 msgr2=0x56150590ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x56150590ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 -- 172.21.15.23:0/1509906735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5801e040 con 0x56150575e6b0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b62ffd700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8b5409ae90 tx=0x7f8b540b3f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58c451061e681bf server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.189+0000 7f8b42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540b2070 con 0x56150575e6b0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f8b5409a890 con 0x56150575e6b0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615058f5710 con 0x56150575e6b0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b540b3b30 con 0x56150575e6b0 2023-11-17T00:04:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 10 client.25574 fetch_fsmap learned FSMap version 26 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56150581a130 con 0x56150575e6b0 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615058f5710 con 0x56150575e6b0 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8b54096f80 con 0x56150575e6b0 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 10 client.25574 fetch_fsmap finished waiting for FSMap version 26 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 10 client.25574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 20 client.25574 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56150596b9d0 0x56150596ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 4 client.25574 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.190+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5615056a2f10 con 0x56150596b9d0 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.191+0000 7f8b627fc700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56150596b9d0 0x56150596ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.191+0000 7f8b42ffd700 10 client.25574 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:17.191+0000 7f8b627fc700 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56150596b9d0 0x56150596ddb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb57f9f2cc58b8aa server_cookie=171ce66a784373a5 in_seq=0 out_seq=0 2023-11-17T00:04:17.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:17 smithi023 sudo[196920]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:17.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:17 smithi023 sudo[196920]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:17.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:17 smithi023 sudo[196920]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:17.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:04:17 smithi023 sudo[196920]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:17.706 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158750]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:17.706 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:17.706 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158750]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:17.706 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158750]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:18.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158753]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:18.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:18.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158753]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:18.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:04:17 smithi156 sudo[158753]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:18.268 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:17 smithi173 sudo[160038]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:18.269 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:17 smithi173 sudo[160038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:18.269 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:17 smithi173 sudo[160038]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:18.269 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:17 smithi173 sudo[160038]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:18.269 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:18 smithi173 ceph-mon[131534]: pgmap v465: 129 pgs: 129 active+clean; 85 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 708 op/s 2023-11-17T00:04:18.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:18 smithi023 ceph-mon[97824]: pgmap v465: 129 pgs: 129 active+clean; 85 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 708 op/s 2023-11-17T00:04:18.606 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:18 smithi173 sudo[160042]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:18.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:18 smithi173 sudo[160042]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:18.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:18 smithi173 sudo[160042]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:18.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:04:18 smithi173 sudo[160042]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:18 smithi156 ceph-mon[131105]: pgmap v465: 129 pgs: 129 active+clean; 85 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 708 op/s 2023-11-17T00:04:19.227 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:18 smithi023 sudo[196971]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:19.227 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:18 smithi023 sudo[196971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:19.227 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:18 smithi023 sudo[196971]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:19.227 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:19 smithi023 sudo[196971]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:19 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:19.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:19 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:19.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:19 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.286+0000 7f8b42ffd700 1 -- 172.21.15.23:0/1509906735 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5615056a2f10 con 0x56150596b9d0 2023-11-17T00:04:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.286+0000 7f8b42ffd700 10 client.25574 handle_command_reply: tid=0 2023-11-17T00:04:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.286+0000 7f8b7d281b80 1 client.25574 shutdown 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 msgr2=0x56150590e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8b5409ae90 tx=0x7f8b540b3f30 comp rx=0 tx=0).stop 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 shutdown_connections 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561505903b10 0x561505911e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56150575e6b0 0x56150590e8f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x56150596b9d0 0x56150596ddb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 --2- 172.21.15.23:0/1509906735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561505902db0 0x56150590ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 >> 172.21.15.23:0/1509906735 conn(0x5615058f7f50 msgr2=0x5615058fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.287+0000 7f8b42ffd700 0 client.25574 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.288+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 shutdown_connections 2023-11-17T00:04:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.288+0000 7f8b7d281b80 1 -- 172.21.15.23:0/1509906735 wait complete. 2023-11-17T00:04:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.288+0000 7f8b7d281b80 20 client.25574 trim_cache size 0 max 16384 2023-11-17T00:04:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:04:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:04:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:04:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.289+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 msgr2=0x7f8b4c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.289+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 0x7f8b4c0a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009ec0 tx=0x7f8b5c0134a0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.289+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 msgr2=0x7f8b640748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.289+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8b54004730 tx=0x7f8b54004760 comp rx=0 tx=0).stop 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 shutdown_connections 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b64101dc0 0x7f8b64073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8b4c0a1150 0x7f8b4c0a3600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b64100240 0x7f8b640748a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 --2- 172.21.15.23:0/2731280822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b64101430 0x7f8b64072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 >> 172.21.15.23:0/2731280822 conn(0x7f8b640fb840 msgr2=0x7f8b64104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 shutdown_connections 2023-11-17T00:04:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:20.290+0000 7f8b699ff700 1 -- 172.21.15.23:0/2731280822 wait complete. 2023-11-17T00:04:20.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:20 smithi023 ceph-mon[97824]: pgmap v466: 129 pgs: 129 active+clean; 85 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2023-11-17T00:04:20.519 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:20 smithi023 sudo[196990]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:20.519 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:20 smithi023 sudo[196990]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:20.519 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:20 smithi023 sudo[196990]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:20.519 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:04:20 smithi023 sudo[196990]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:20 smithi173 ceph-mon[131534]: pgmap v466: 129 pgs: 129 active+clean; 85 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2023-11-17T00:04:20.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:20 smithi156 ceph-mon[131105]: pgmap v466: 129 pgs: 129 active+clean; 85 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 291 op/s 2023-11-17T00:04:20.880 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158758]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:20.881 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158758]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:20.881 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158758]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:20.881 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158758]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:21.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158761]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:21.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158761]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:21.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158761]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:21.357 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:04:20 smithi156 sudo[158761]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:21.457 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160047]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:21.458 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160047]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:21.458 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160047]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:21.458 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160047]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:21.857 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160050]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:21.857 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:21.857 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160050]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:21.857 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:04:21 smithi173 sudo[160050]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198367]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198367]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198367]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198367]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:23.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198405]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:23.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:23.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198405]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:23.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:22 smithi023 sudo[198405]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:23.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:22 smithi173 sudo[160054]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:23.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:22 smithi173 sudo[160054]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:23.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:22 smithi173 sudo[160054]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:23.260 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:22 smithi173 sudo[160054]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: pgmap v467: 129 pgs: 129 active+clean; 90 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 319 op/s 2023-11-17T00:04:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:23 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: pgmap v467: 129 pgs: 129 active+clean; 90 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 319 op/s 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158768]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-17T00:04:23.527 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:23.527 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158768]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:23.527 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158768]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 sudo[160058]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 sudo[160058]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 sudo[160058]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 sudo[160058]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: pgmap v467: 129 pgs: 129 active+clean; 90 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 319 op/s 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:23.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:23.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:23 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:23.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158771]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-17T00:04:23.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158771]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-17T00:04:23.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158771]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-17T00:04:23.857 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:23 smithi156 sudo[158771]: pam_unix(sudo:session): session closed for user root 2023-11-17T00:04:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: pgmap v468: 129 pgs: 129 active+clean; 100 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 783 op/s 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:24 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:04:24.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: pgmap v468: 129 pgs: 129 active+clean; 100 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 783 op/s 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:24 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: pgmap v468: 129 pgs: 129 active+clean; 100 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 783 op/s 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-17T00:04:24.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:24 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:04:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: pgmap v469: 129 pgs: 129 active+clean; 91 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 1.11k op/s 2023-11-17T00:04:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:04:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:04:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:04:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:26 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:04:26.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: pgmap v469: 129 pgs: 129 active+clean; 91 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 1.11k op/s 2023-11-17T00:04:26.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:04:26.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:04:26.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:26.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:04:26.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:26 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:04:26.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: pgmap v469: 129 pgs: 129 active+clean; 91 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 1.11k op/s 2023-11-17T00:04:26.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:04:26.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:04:26.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:04:26.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:04:26.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:04:26.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:26 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:04:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:28 smithi173 ceph-mon[131534]: pgmap v470: 129 pgs: 129 active+clean; 93 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 1.08k op/s 2023-11-17T00:04:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:28 smithi156 ceph-mon[131105]: pgmap v470: 129 pgs: 129 active+clean; 93 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 1.08k op/s 2023-11-17T00:04:28.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:28 smithi023 ceph-mon[97824]: pgmap v470: 129 pgs: 129 active+clean; 93 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 1.08k op/s 2023-11-17T00:04:30.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:29 smithi173 ceph-mon[131534]: pgmap v471: 129 pgs: 129 active+clean; 99 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 1.08k op/s 2023-11-17T00:04:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:29 smithi156 ceph-mon[131105]: pgmap v471: 129 pgs: 129 active+clean; 99 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 1.08k op/s 2023-11-17T00:04:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:29 smithi023 ceph-mon[97824]: pgmap v471: 129 pgs: 129 active+clean; 99 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 1.08k op/s 2023-11-17T00:04:31.661 INFO:tasks.workunit.client.0.smithi023.stdout:created hierarchy, now cleaning up 2023-11-17T00:04:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:32 smithi023 ceph-mon[97824]: pgmap v472: 129 pgs: 129 active+clean; 106 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-17T00:04:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:32 smithi173 ceph-mon[131534]: pgmap v472: 129 pgs: 129 active+clean; 106 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-17T00:04:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:32 smithi156 ceph-mon[131105]: pgmap v472: 129 pgs: 129 active+clean; 106 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 1.06k op/s 2023-11-17T00:04:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 -- 172.21.15.23:0/3432957361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4338100cc0 msgr2=0x7f43381030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 --2- 172.21.15.23:0/3432957361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4338100cc0 0x7f43381030a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f432c009990 tx=0x7f432c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 -- 172.21.15.23:0/3432957361 shutdown_connections 2023-11-17T00:04:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 --2- 172.21.15.23:0/3432957361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f4338108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 --2- 172.21.15.23:0/3432957361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4338100cc0 0x7f43381030a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 --2- 172.21.15.23:0/3432957361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43381035e0 0x7f43381059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.447+0000 7f433edff700 1 -- 172.21.15.23:0/3432957361 >> 172.21.15.23:0/3432957361 conn(0x7f43380fa720 msgr2=0x7f43380fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.448+0000 7f433edff700 1 -- 172.21.15.23:0/3432957361 shutdown_connections 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.448+0000 7f433edff700 1 -- 172.21.15.23:0/3432957361 wait complete. 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.448+0000 7f433edff700 1 Processor -- start 2023-11-17T00:04:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 -- start start 2023-11-17T00:04:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 0x7f4338195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f433819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433819db50 con 0x7f4338105f00 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433819dc90 con 0x7f43381035e0 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433edff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433819de20 con 0x7f4338100cc0 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 0x7f4338195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f4337fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 0x7f4338195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:42372/0 (socket says 172.21.15.23:42372) 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f4337fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:55580/0 (socket says 172.21.15.23:55580) 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433cb9b700 1 -- 172.21.15.23:0/1665243025 learned_addr learned my addr 172.21.15.23:0/1665243025 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.449+0000 7f433d39c700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f433819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 -- 172.21.15.23:0/1665243025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 msgr2=0x7f4338195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 0x7f4338195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 -- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 msgr2=0x7f433819d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f433819d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 -- 172.21.15.23:0/1665243025 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c0095f0 con 0x7f43381035e0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f433d39c700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f433819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4337fff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f432400e9f0 tx=0x7f432400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4335ffb700 1 -- 172.21.15.23:0/1665243025 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432400eee0 con 0x7f43381035e0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433810bbf0 con 0x7f43381035e0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4335ffb700 1 -- 172.21.15.23:0/1665243025 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f432400cbd0 con 0x7f43381035e0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f4335ffb700 1 -- 172.21.15.23:0/1665243025 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324018680 con 0x7f43381035e0 2023-11-17T00:04:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.450+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433810ba60 con 0x7f43381035e0 2023-11-17T00:04:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.454+0000 7f4335ffb700 1 -- 172.21.15.23:0/1665243025 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f432400cd40 con 0x7f43381035e0 2023-11-17T00:04:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 Processor -- start 2023-11-17T00:04:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4335ffb700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 0x7f43280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 -- start start 2023-11-17T00:04:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4335ffb700 1 -- 172.21.15.23:0/1665243025 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4324014070 con 0x7f43381035e0 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5e4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f4e9000 con 0x55d40f5e3db0 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f53b840 con 0x55d40f43f6b0 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.455+0000 7f4352779b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f534cd0 con 0x55d40f5e4b10 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433d39c700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 0x7f43280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:55586/0 (socket says 172.21.15.23:55586) 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 -- 172.21.15.23:0/1811000663 learned_addr learned my addr 172.21.15.23:0/1811000663 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f4337fff700 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 -- 172.21.15.23:0/1811000663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 msgr2=0x55d40f5e4f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5e4f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 -- 172.21.15.23:0/1811000663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 msgr2=0x55d40f5e45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5e45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 -- 172.21.15.23:0/1811000663 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d40f3b65d0 con 0x55d40f43f6b0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433cb9b700 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f432c005f30 tx=0x7f432c005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eadc633caaa13c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.456+0000 7f433d39c700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 0x7f43280a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4330009450 tx=0x7f433000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4322ffd700 1 -- 172.21.15.23:0/1811000663 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c01d070 con 0x55d40f43f6b0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4322ffd700 1 -- 172.21.15.23:0/1811000663 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f432c01e040 con 0x55d40f43f6b0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4322ffd700 1 -- 172.21.15.23:0/1811000663 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c004a60 con 0x55d40f43f6b0 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 -- 172.21.15.23:0/1811000663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 msgr2=0x55d40f5e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f432c005f30 tx=0x7f432c005bc0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 -- 172.21.15.23:0/1811000663 shutdown_connections 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5e4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5e3870 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 --2- 172.21.15.23:0/1811000663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5e45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.457+0000 7f4352779b80 1 -- 172.21.15.23:0/1811000663 >> 172.21.15.23:0/1811000663 conn(0x55d40f5d8f50 msgr2=0x55d40f5de100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.458+0000 7f4352779b80 1 -- 172.21.15.23:0/1811000663 shutdown_connections 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.458+0000 7f4352779b80 1 -- 172.21.15.23:0/1811000663 wait complete. 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.458+0000 7f4352779b80 1 -- start start 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.458+0000 7f4352779b80 1 Processor -- start 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e4b10 0x55d40f5f8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f4d6880 con 0x55d40f5e3db0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f4d6520 con 0x55d40f5e4b10 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f3b88c0 con 0x55d40f43f6b0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4352779b80 10 client.0 _ll_register_callbacks cb 0x55d40f4f4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433d39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e4b10 0x55d40f5f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:42386/0 (socket says 172.21.15.23:42386) 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f4337fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 -- 172.21.15.23:0/2517487648 learned_addr learned my addr 172.21.15.23:0/2517487648 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 -- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e4b10 msgr2=0x55d40f5f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e4b10 0x55d40f5f8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 -- 172.21.15.23:0/2517487648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 msgr2=0x55d40f5f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.459+0000 7f433cb9b700 1 -- 172.21.15.23:0/2517487648 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4338108f50 con 0x55d40f43f6b0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f433cb9b700 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f432c0057d0 tx=0x7f432c022630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c14e49f0f0833e3 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4317fff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c01d070 con 0x55d40f43f6b0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d40f5d6710 con 0x55d40f43f6b0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f432c022cf0 con 0x55d40f43f6b0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f432c00f460 con 0x55d40f43f6b0 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4352779b80 10 client.25621 fetch_fsmap learned FSMap version 27 2023-11-17T00:04:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d40f4fb130 con 0x55d40f43f6b0 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.460+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d40f5d6710 con 0x55d40f43f6b0 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f432c02c5c0 con 0x55d40f43f6b0 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 10 client.25621 fetch_fsmap finished waiting for FSMap version 27 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 10 client.25621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 20 client.25621 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f5fca20 0x55d40f5fee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 4 client.25621 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d40f4f7040 con 0x55d40f5fca20 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4337fff700 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f5fca20 0x55d40f5fee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4337fff700 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f5fca20 0x55d40f5fee30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=856e3086f4b1706f server_cookie=aa430c05868b05ed in_seq=0 out_seq=0 2023-11-17T00:04:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.461+0000 7f4317fff700 10 client.25621 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.465+0000 7f4317fff700 1 -- 172.21.15.23:0/2517487648 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d40f4f7040 con 0x55d40f5fca20 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.465+0000 7f4317fff700 10 client.25621 handle_command_reply: tid=0 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 client.25621 shutdown 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 msgr2=0x55d40f5f1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f432c0057d0 tx=0x7f432c022630 comp rx=0 tx=0).stop 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 shutdown_connections 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e4b10 0x55d40f5f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f1080 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f5fca20 0x55d40f5fee30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 --2- 172.21.15.23:0/2517487648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 >> 172.21.15.23:0/2517487648 conn(0x55d40f5d8f50 msgr2=0x55d40f5dd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:33.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4317fff700 0 client.25621 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:33.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 shutdown_connections 2023-11-17T00:04:33.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 1 -- 172.21.15.23:0/2517487648 wait complete. 2023-11-17T00:04:33.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.466+0000 7f4352779b80 20 client.25621 trim_cache size 0 max 16384 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 Processor -- start 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 -- start start 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5da420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5da960 con 0x55d40f5e3db0 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5ecb00 con 0x55d40f43f6b0 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4352779b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5df930 con 0x55d40f5e4b10 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4337fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f433d39c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4337fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39424/0 (socket says 172.21.15.23:39424) 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:55614/0 (socket says 172.21.15.23:55614) 2023-11-17T00:04:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.482+0000 7f4337fff700 1 -- 172.21.15.23:0/166210627 learned_addr learned my addr 172.21.15.23:0/166210627 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 -- 172.21.15.23:0/166210627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 msgr2=0x55d40f5f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 -- 172.21.15.23:0/166210627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 msgr2=0x55d40f5f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 -- 172.21.15.23:0/166210627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d40f4fae80 con 0x55d40f5e4b10 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433cb9b700 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f4337fff700 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f433d39c700 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5da420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4330009fd0 tx=0x7f4330013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41e4805b79bd3723 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.483+0000 7f43237fe700 1 -- 172.21.15.23:0/166210627 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000e050 con 0x55d40f5e4b10 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f43237fe700 1 -- 172.21.15.23:0/166210627 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f433001a070 con 0x55d40f5e4b10 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f43237fe700 1 -- 172.21.15.23:0/166210627 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330016970 con 0x55d40f5e4b10 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 -- 172.21.15.23:0/166210627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 msgr2=0x55d40f5da420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5da420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4330009fd0 tx=0x7f4330013000 comp rx=0 tx=0).stop 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 -- 172.21.15.23:0/166210627 shutdown_connections 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f5e4b10 0x55d40f5da420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f43f6b0 0x55d40f5f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 --2- 172.21.15.23:0/166210627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e3db0 0x55d40f5f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 -- 172.21.15.23:0/166210627 >> 172.21.15.23:0/166210627 conn(0x55d40f5d8f50 msgr2=0x55d40f603750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 -- 172.21.15.23:0/166210627 shutdown_connections 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.484+0000 7f4352779b80 1 -- 172.21.15.23:0/166210627 wait complete. 2023-11-17T00:04:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 -- start start 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 Processor -- start 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 0x55d40f5f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e4b10 0x55d40f5f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5f78f0 con 0x55d40f5e4b10 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5f7a60 con 0x55d40f5e3db0 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40f5f7c20 con 0x55d40f43f6b0 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f4352779b80 10 client.0 _ll_register_callbacks cb 0x7f4324010ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.485+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 0x55d40f5f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f433cb9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:42414/0 (socket says 172.21.15.23:42414) 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f433cb9b700 1 -- 172.21.15.23:0/3411378459 learned_addr learned my addr 172.21.15.23:0/3411378459 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f433d39c700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e4b10 0x55d40f5f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 -- 172.21.15.23:0/3411378459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 msgr2=0x55d40f5f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 -- 172.21.15.23:0/3411378459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e4b10 msgr2=0x55d40f5f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e4b10 0x55d40f5f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 -- 172.21.15.23:0/3411378459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d40f4fb130 con 0x55d40f5e3db0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f433cb9b700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4337fff700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 0x55d40f5f6ca0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4324096970 tx=0x7f4324099b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bdb2aceb443b1a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4317fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240b82f0 con 0x55d40f5e3db0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f432c01e040 con 0x55d40f5e3db0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.486+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f43240993f0 con 0x55d40f5e3db0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43240995b0 con 0x55d40f5e3db0 2023-11-17T00:04:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 10 client.25595 fetch_fsmap learned FSMap version 27 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f433810ba60 con 0x55d40f5e3db0 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4324063080 con 0x55d40f5e3db0 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f4324095cf0 con 0x55d40f5e3db0 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 10 client.25595 fetch_fsmap finished waiting for FSMap version 27 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 10 client.25595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 20 client.25595 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f6513d0 0x55d40f6537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 4 client.25595 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d40f503600 con 0x55d40f6513d0 2023-11-17T00:04:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.487+0000 7f433cb9b700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f6513d0 0x55d40f6537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.488+0000 7f433cb9b700 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f6513d0 0x55d40f6537e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2165537881c59d server_cookie=122e277b5f8bf080 in_seq=0 out_seq=0 2023-11-17T00:04:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:33.488+0000 7f4317fff700 10 client.25595 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:34 smithi023 ceph-mon[97824]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 845 op/s 2023-11-17T00:04:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:34 smithi173 ceph-mon[131534]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 845 op/s 2023-11-17T00:04:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:34 smithi156 ceph-mon[131105]: pgmap v473: 129 pgs: 129 active+clean; 116 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 845 op/s 2023-11-17T00:04:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:36 smithi023 ceph-mon[97824]: pgmap v474: 129 pgs: 129 active+clean; 121 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2023-11-17T00:04:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:36 smithi173 ceph-mon[131534]: pgmap v474: 129 pgs: 129 active+clean; 121 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2023-11-17T00:04:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:36 smithi156 ceph-mon[131105]: pgmap v474: 129 pgs: 129 active+clean; 121 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4317fff700 1 -- 172.21.15.23:0/3411378459 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d40f503600 con 0x55d40f6513d0 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4317fff700 10 client.25595 handle_command_reply: tid=0 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 client.25595 shutdown 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 msgr2=0x55d40f5f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 0x55d40f5f6ca0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4324096970 tx=0x7f4324099b60 comp rx=0 tx=0).stop 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 shutdown_connections 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d40f6513d0 0x55d40f6537e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d40f5e4b10 0x55d40f5f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d40f43f6b0 0x55d40f5f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4317fff700 0 client.25595 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 --2- 172.21.15.23:0/3411378459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d40f5e3db0 0x55d40f5f6ca0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 >> 172.21.15.23:0/3411378459 conn(0x55d40f5d8f50 msgr2=0x55d40f5f05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 shutdown_connections 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 1 -- 172.21.15.23:0/3411378459 wait complete. 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.669+0000 7f4352779b80 20 client.25595 trim_cache size 0 max 16384 2023-11-17T00:04:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:04:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:04:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:04:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:04:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 msgr2=0x7f43280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 0x7f43280a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4330009450 tx=0x7f433000b040 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 msgr2=0x7f4338196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f432400e9f0 tx=0x7f432400ed00 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 shutdown_connections 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4338105f00 0x7f433819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f43280a1150 0x7f43280a3600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4338100cc0 0x7f4338195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 --2- 172.21.15.23:0/1665243025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43381035e0 0x7f4338196510 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 >> 172.21.15.23:0/1665243025 conn(0x7f43380fa720 msgr2=0x7f4338102120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 shutdown_connections 2023-11-17T00:04:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:37.670+0000 7f433edff700 1 -- 172.21.15.23:0/1665243025 wait complete. 2023-11-17T00:04:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:38 smithi023 ceph-mon[97824]: pgmap v475: 129 pgs: 129 active+clean; 122 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 47 op/s 2023-11-17T00:04:38.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:38 smithi173 ceph-mon[131534]: pgmap v475: 129 pgs: 129 active+clean; 122 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 47 op/s 2023-11-17T00:04:38.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:38 smithi156 ceph-mon[131105]: pgmap v475: 129 pgs: 129 active+clean; 122 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 47 op/s 2023-11-17T00:04:40.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:39 smithi173 ceph-mon[131534]: pgmap v476: 129 pgs: 129 active+clean; 121 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2023-11-17T00:04:40.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:39 smithi156 ceph-mon[131105]: pgmap v476: 129 pgs: 129 active+clean; 121 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2023-11-17T00:04:40.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:39 smithi023 ceph-mon[97824]: pgmap v476: 129 pgs: 129 active+clean; 121 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2023-11-17T00:04:42.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:41 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:42.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:41 smithi023 ceph-mon[97824]: pgmap v477: 129 pgs: 129 active+clean; 119 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2023-11-17T00:04:42.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:41 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:41 smithi173 ceph-mon[131534]: pgmap v477: 129 pgs: 129 active+clean; 119 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2023-11-17T00:04:42.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:41 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:42.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:41 smithi156 ceph-mon[131105]: pgmap v477: 129 pgs: 129 active+clean; 119 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 302 op/s 2023-11-17T00:04:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:44 smithi023 ceph-mon[97824]: pgmap v478: 129 pgs: 129 active+clean; 119 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 432 op/s 2023-11-17T00:04:44.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:44 smithi173 ceph-mon[131534]: pgmap v478: 129 pgs: 129 active+clean; 119 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 432 op/s 2023-11-17T00:04:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:44 smithi156 ceph-mon[131105]: pgmap v478: 129 pgs: 129 active+clean; 119 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 432 op/s 2023-11-17T00:04:46.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:46 smithi023 ceph-mon[97824]: pgmap v479: 129 pgs: 129 active+clean; 124 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 467 op/s 2023-11-17T00:04:46.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:46 smithi173 ceph-mon[131534]: pgmap v479: 129 pgs: 129 active+clean; 124 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 467 op/s 2023-11-17T00:04:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:46 smithi156 ceph-mon[131105]: pgmap v479: 129 pgs: 129 active+clean; 124 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 467 op/s 2023-11-17T00:04:47.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:47 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:47.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:47 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:47.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:47 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:04:48.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:48 smithi023 ceph-mon[97824]: pgmap v480: 129 pgs: 129 active+clean; 125 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 465 op/s 2023-11-17T00:04:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:48 smithi173 ceph-mon[131534]: pgmap v480: 129 pgs: 129 active+clean; 125 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 465 op/s 2023-11-17T00:04:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:48 smithi156 ceph-mon[131105]: pgmap v480: 129 pgs: 129 active+clean; 125 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 465 op/s 2023-11-17T00:04:50.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:49 smithi173 ceph-mon[131534]: pgmap v481: 129 pgs: 129 active+clean; 129 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 504 op/s 2023-11-17T00:04:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:49 smithi156 ceph-mon[131105]: pgmap v481: 129 pgs: 129 active+clean; 129 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 504 op/s 2023-11-17T00:04:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:49 smithi023 ceph-mon[97824]: pgmap v481: 129 pgs: 129 active+clean; 129 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 504 op/s 2023-11-17T00:04:50.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 -- 172.21.15.23:0/3420993158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c074ab0 msgr2=0x7f501c072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 --2- 172.21.15.23:0/3420993158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c074ab0 0x7f501c072f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f50180098f0 tx=0x7f5018009c00 comp rx=0 tx=0).stop 2023-11-17T00:04:50.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 -- 172.21.15.23:0/3420993158 shutdown_connections 2023-11-17T00:04:50.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 --2- 172.21.15.23:0/3420993158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c1080f0 0x7f501c10a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 --2- 172.21.15.23:0/3420993158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c074ab0 0x7f501c072f00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 --2- 172.21.15.23:0/3420993158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c0734d0 0x7f501c073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 -- 172.21.15.23:0/3420993158 >> 172.21.15.23:0/3420993158 conn(0x7f501c0fb840 msgr2=0x7f501c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:50.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 -- 172.21.15.23:0/3420993158 shutdown_connections 2023-11-17T00:04:50.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.843+0000 7f50237ff700 1 -- 172.21.15.23:0/3420993158 wait complete. 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.844+0000 7f50237ff700 1 Processor -- start 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 -- start start 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c074ab0 0x7f501c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c1080f0 0x7f501c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c19fdd0 con 0x7f501c074ab0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c19ff10 con 0x7f501c0734d0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f50237ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c1a00a0 con 0x7f501c1080f0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47818/0 (socket says 172.21.15.23:47818) 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f502159b700 1 -- 172.21.15.23:0/2289079429 learned_addr learned my addr 172.21.15.23:0/2289079429 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f5021d9c700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c1080f0 0x7f501c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.845+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c074ab0 0x7f501c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 -- 172.21.15.23:0/2289079429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c1080f0 msgr2=0x7f501c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c1080f0 0x7f501c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 -- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c074ab0 msgr2=0x7f501c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c074ab0 0x7f501c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 -- 172.21.15.23:0/2289079429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c009520 con 0x7f501c0734d0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f502159b700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5018009fd0 tx=0x7f501800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f50127fc700 1 -- 172.21.15.23:0/2289079429 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501801c070 con 0x7f501c0734d0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5014009d00 con 0x7f501c0734d0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f50127fc700 1 -- 172.21.15.23:0/2289079429 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f501800f7d0 con 0x7f501c0734d0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f50127fc700 1 -- 172.21.15.23:0/2289079429 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018004b90 con 0x7f501c0734d0 2023-11-17T00:04:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.846+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5018009520 con 0x7f501c0734d0 2023-11-17T00:04:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.850+0000 7f50127fc700 1 -- 172.21.15.23:0/2289079429 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5018020070 con 0x7f501c0734d0 2023-11-17T00:04:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.852+0000 7f50371deb80 1 Processor -- start 2023-11-17T00:04:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.852+0000 7f50371deb80 1 -- start start 2023-11-17T00:04:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e6476b0 0x55609e7f4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7f5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50127fc700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 0x7f50080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e6f1000 con 0x55609e7f5320 2023-11-17T00:04:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e743840 con 0x55609e7f45c0 2023-11-17T00:04:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50371deb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e73ccd0 con 0x55609e6476b0 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f5021d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e6476b0 0x55609e7f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47822/0 (socket says 172.21.15.23:47822) 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 -- 172.21.15.23:0/2433197938 learned_addr learned my addr 172.21.15.23:0/2433197938 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f50127fc700 1 -- 172.21.15.23:0/2289079429 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f50180cd960 con 0x7f501c0734d0 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f5021d9c700 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 0x7f50080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 -- 172.21.15.23:0/2433197938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e6476b0 msgr2=0x55609e7f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e6476b0 0x55609e7f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 -- 172.21.15.23:0/2433197938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 msgr2=0x55609e7f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.853+0000 7f502159b700 1 -- 172.21.15.23:0/2433197938 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55609e5be5d0 con 0x55609e7f45c0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.854+0000 7f502159b700 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f50180cb0b0 tx=0x7f5018009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a05b94a937a6408 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.854+0000 7f50077fe700 1 -- 172.21.15.23:0/2433197938 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501809a590 con 0x55609e7f45c0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.854+0000 7f50077fe700 1 -- 172.21.15.23:0/2433197938 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f501809a3b0 con 0x55609e7f45c0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.854+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 0x7f50080a3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d80 tx=0x7f500c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.854+0000 7f50077fe700 1 -- 172.21.15.23:0/2433197938 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180a08e0 con 0x55609e7f45c0 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- 172.21.15.23:0/2433197938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 msgr2=0x55609e7f4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f50180cb0b0 tx=0x7f5018009c00 comp rx=0 tx=0).stop 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- 172.21.15.23:0/2433197938 shutdown_connections 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e6476b0 0x55609e7f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 --2- 172.21.15.23:0/2433197938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e7f45c0 0x55609e7f4de0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- 172.21.15.23:0/2433197938 >> 172.21.15.23:0/2433197938 conn(0x55609e7e0f50 msgr2=0x55609e7e6100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- 172.21.15.23:0/2433197938 shutdown_connections 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- 172.21.15.23:0/2433197938 wait complete. 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.855+0000 7f50371deb80 1 -- start start 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.856+0000 7f50371deb80 1 Processor -- start 2023-11-17T00:04:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.856+0000 7f50371deb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e7fd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e804660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e6de880 con 0x55609e7f5320 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e6de520 con 0x55609e6476b0 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e5c08c0 con 0x55609e7f45c0 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f50371deb80 10 client.0 _ll_register_callbacks cb 0x55609e6fc720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47834/0 (socket says 172.21.15.23:47834) 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e7fd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f502159b700 1 -- 172.21.15.23:0/4046630557 learned_addr learned my addr 172.21.15.23:0/4046630557 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 -- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 msgr2=0x55609e7fd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 -- 172.21.15.23:0/4046630557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 msgr2=0x55609e804660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e804660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.857+0000 7f5020d9a700 1 -- 172.21.15.23:0/4046630557 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5018009520 con 0x55609e7f45c0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.858+0000 7f5020d9a700 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e7fd620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f500c011a50 tx=0x7f500c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34e54fd8222555fe server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.858+0000 7f5004ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.858+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c004a30 con 0x55609e7f45c0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.858+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f500c018070 con 0x55609e7f45c0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.858+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55609e7de710 con 0x55609e7f45c0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f500c004b90 con 0x55609e7f45c0 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 10 client.25639 fetch_fsmap learned FSMap version 29 2023-11-17T00:04:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55609e703130 con 0x55609e7f45c0 2023-11-17T00:04:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55609e7de710 con 0x55609e7f45c0 2023-11-17T00:04:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f500c025070 con 0x55609e7f45c0 2023-11-17T00:04:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 10 client.25639 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:04:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 20 client.25639 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e809020 0x55609e80b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 4 client.25639 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.860+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55609e6ff040 con 0x55609e809020 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.861+0000 7f502159b700 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e809020 0x55609e80b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.861+0000 7f502159b700 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e809020 0x55609e80b430 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f49442a6910cbb6 server_cookie=73077c3f14bfe572 in_seq=0 out_seq=0 2023-11-17T00:04:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.861+0000 7f5004ff9700 10 client.25639 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.863+0000 7f5004ff9700 1 -- 172.21.15.23:0/4046630557 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55609e6ff040 con 0x55609e809020 2023-11-17T00:04:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.863+0000 7f5004ff9700 10 client.25639 handle_command_reply: tid=0 2023-11-17T00:04:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.863+0000 7f50371deb80 1 client.25639 shutdown 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 msgr2=0x55609e7fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e7fd620 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f500c011a50 tx=0x7f500c0047c0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 shutdown_connections 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e809020 0x55609e80b430 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e804660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 --2- 172.21.15.23:0/4046630557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e7fd620 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 >> 172.21.15.23:0/4046630557 conn(0x55609e7e0f50 msgr2=0x55609e7e57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.864+0000 7f5004ff9700 0 client.25639 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.865+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 shutdown_connections 2023-11-17T00:04:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.865+0000 7f50371deb80 1 -- 172.21.15.23:0/4046630557 wait complete. 2023-11-17T00:04:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.865+0000 7f50371deb80 20 client.25639 trim_cache size 0 max 16384 2023-11-17T00:04:50.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.887+0000 7f50371deb80 1 Processor -- start 2023-11-17T00:04:50.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.887+0000 7f50371deb80 1 -- start start 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e804620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7fd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e7e0b90 con 0x55609e7f5320 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e7f4990 con 0x55609e6476b0 2023-11-17T00:04:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.888+0000 7f50371deb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e7fdd60 con 0x55609e7f45c0 2023-11-17T00:04:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47836/0 (socket says 172.21.15.23:47836) 2023-11-17T00:04:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f502159b700 1 -- 172.21.15.23:0/3873696465 learned_addr learned my addr 172.21.15.23:0/3873696465 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f5021d9c700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7fd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f5020d9a700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e804620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.889+0000 7f502159b700 1 -- 172.21.15.23:0/3873696465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 msgr2=0x55609e804620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.890+0000 7f502159b700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e804620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.890+0000 7f502159b700 1 -- 172.21.15.23:0/3873696465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 msgr2=0x55609e7fd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.890+0000 7f502159b700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7fd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.890+0000 7f502159b700 1 -- 172.21.15.23:0/3873696465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55609e702e80 con 0x55609e6476b0 2023-11-17T00:04:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.890+0000 7f5021d9c700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7fd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.891+0000 7f502159b700 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5018096f50 tx=0x7f50180b3280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddb26d9ae3ad73b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.892+0000 7f5007fff700 1 -- 172.21.15.23:0/3873696465 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501809ea80 con 0x55609e6476b0 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.892+0000 7f5007fff700 1 -- 172.21.15.23:0/3873696465 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5018098df0 con 0x55609e6476b0 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 -- 172.21.15.23:0/3873696465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 msgr2=0x55609e7fce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5018096f50 tx=0x7f50180b3280 comp rx=0 tx=0).stop 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 -- 172.21.15.23:0/3873696465 shutdown_connections 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e7fd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e7fce90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 --2- 172.21.15.23:0/3873696465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e804620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.893+0000 7f50371deb80 1 -- 172.21.15.23:0/3873696465 >> 172.21.15.23:0/3873696465 conn(0x55609e7e0f50 msgr2=0x55609e80fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.894+0000 7f50371deb80 1 -- 172.21.15.23:0/3873696465 shutdown_connections 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.894+0000 7f50371deb80 1 -- 172.21.15.23:0/3873696465 wait complete. 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.895+0000 7f50371deb80 1 -- start start 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.896+0000 7f50371deb80 1 Processor -- start 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e85b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e85ba80 con 0x55609e7f5320 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e85bbf0 con 0x55609e6476b0 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55609e85bdb0 con 0x55609e7f45c0 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.898+0000 7f50371deb80 10 client.0 _ll_register_callbacks cb 0x55609e6fc720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.899+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.899+0000 7f502159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:47844/0 (socket says 172.21.15.23:47844) 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.899+0000 7f5020d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.899+0000 7f5020d9a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:40044/0 (socket says 172.21.15.23:40044) 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.899+0000 7f5020d9a700 1 -- 172.21.15.23:0/2695672423 learned_addr learned my addr 172.21.15.23:0/2695672423 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.900+0000 7f5021d9c700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e85b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.901+0000 7f5020d9a700 1 -- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 msgr2=0x55609e802570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.901+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.901+0000 7f5020d9a700 1 -- 172.21.15.23:0/2695672423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 msgr2=0x55609e85b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.901+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e85b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.902+0000 7f5020d9a700 1 -- 172.21.15.23:0/2695672423 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55609e5be5d0 con 0x55609e7f45c0 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.902+0000 7f5021d9c700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e85b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.902+0000 7f502159b700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.902+0000 7f5020d9a700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f500c0093d0 tx=0x7f500c0276e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd02a24f7bee90b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:04:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.902+0000 7f5004ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.903+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c027d30 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.903+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f500c018070 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.903+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55609e70b600 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.904+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f500c027e90 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.905+0000 7f50371deb80 10 client.25645 fetch_fsmap learned FSMap version 29 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.905+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55609e553c00 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55609e70b600 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f500c012870 con 0x55609e7f45c0 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 10 client.25645 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 20 client.25645 populate_metadata read hostname 'smithi023' 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e85d790 0x55609e85fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:04:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 4 client.25645 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:04:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.906+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55609e58bf10 con 0x55609e85d790 2023-11-17T00:04:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.907+0000 7f502159b700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e85d790 0x55609e85fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:04:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.910+0000 7f502159b700 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e85d790 0x55609e85fba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=670acc43cfa747f3 server_cookie=29182bfb3f80e24 in_seq=0 out_seq=0 2023-11-17T00:04:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:50.910+0000 7f5004ff9700 10 client.25645 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f5004ff9700 1 -- 172.21.15.23:0/2695672423 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55609e58bf10 con 0x55609e85d790 2023-11-17T00:04:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f5004ff9700 10 client.25645 handle_command_reply: tid=0 2023-11-17T00:04:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 client.25645 shutdown 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 msgr2=0x55609e858e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f500c0093d0 tx=0x7f500c0276e0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 shutdown_connections 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55609e85d790 0x55609e85fba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55609e7f5320 0x55609e85b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609e6476b0 0x55609e802570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f5004ff9700 0 client.25645 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 --2- 172.21.15.23:0/2695672423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55609e7f45c0 0x55609e858e70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 >> 172.21.15.23:0/2695672423 conn(0x55609e7e0f50 msgr2=0x55609e7fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.247+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 shutdown_connections 2023-11-17T00:04:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.248+0000 7f50371deb80 1 -- 172.21.15.23:0/2695672423 wait complete. 2023-11-17T00:04:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.248+0000 7f50371deb80 20 client.25645 trim_cache size 0 max 16384 2023-11-17T00:04:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:04:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:04:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:04:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 msgr2=0x7f50080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 0x7f50080a3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f500c009d80 tx=0x7f500c009260 comp rx=0 tx=0).stop 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 msgr2=0x7f501c198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5018009fd0 tx=0x7f501800b790 comp rx=0 tx=0).stop 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 shutdown_connections 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f50080a11d0 0x7f50080a3680 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f501c1080f0 0x7f501c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f501c0734d0 0x7f501c198250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 --2- 172.21.15.23:0/2289079429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f501c074ab0 0x7f501c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:04:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 >> 172.21.15.23:0/2289079429 conn(0x7f501c0fb840 msgr2=0x7f501c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:04:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 shutdown_connections 2023-11-17T00:04:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:04:52.249+0000 7f50237ff700 1 -- 172.21.15.23:0/2289079429 wait complete. 2023-11-17T00:04:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:52 smithi023 ceph-mon[97824]: pgmap v482: 129 pgs: 129 active+clean; 143 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2023-11-17T00:04:52.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:52 smithi173 ceph-mon[131534]: pgmap v482: 129 pgs: 129 active+clean; 143 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2023-11-17T00:04:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:52 smithi156 ceph-mon[131105]: pgmap v482: 129 pgs: 129 active+clean; 143 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2023-11-17T00:04:54.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:54 smithi023 ceph-mon[97824]: pgmap v483: 129 pgs: 129 active+clean; 149 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 413 op/s 2023-11-17T00:04:54.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:54 smithi173 ceph-mon[131534]: pgmap v483: 129 pgs: 129 active+clean; 149 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 413 op/s 2023-11-17T00:04:54.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:54 smithi156 ceph-mon[131105]: pgmap v483: 129 pgs: 129 active+clean; 149 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 413 op/s 2023-11-17T00:04:56.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:56 smithi023 ceph-mon[97824]: pgmap v484: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 486 op/s 2023-11-17T00:04:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:56 smithi173 ceph-mon[131534]: pgmap v484: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 486 op/s 2023-11-17T00:04:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:56 smithi156 ceph-mon[131105]: pgmap v484: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 486 op/s 2023-11-17T00:04:58.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:58 smithi023 ceph-mon[97824]: pgmap v485: 129 pgs: 129 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 537 op/s 2023-11-17T00:04:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:58 smithi156 ceph-mon[131105]: pgmap v485: 129 pgs: 129 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 537 op/s 2023-11-17T00:04:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:58 smithi173 ceph-mon[131534]: pgmap v485: 129 pgs: 129 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 537 op/s 2023-11-17T00:05:00.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:04:59 smithi156 ceph-mon[131105]: pgmap v486: 129 pgs: 129 active+clean; 146 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 613 op/s 2023-11-17T00:05:00.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:04:59 smithi173 ceph-mon[131534]: pgmap v486: 129 pgs: 129 active+clean; 146 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 613 op/s 2023-11-17T00:05:00.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:04:59 smithi023 ceph-mon[97824]: pgmap v486: 129 pgs: 129 active+clean; 146 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 613 op/s 2023-11-17T00:05:02.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:02 smithi023 ceph-mon[97824]: pgmap v487: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 670 op/s 2023-11-17T00:05:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:02 smithi173 ceph-mon[131534]: pgmap v487: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 670 op/s 2023-11-17T00:05:02.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:02 smithi156 ceph-mon[131105]: pgmap v487: 129 pgs: 129 active+clean; 154 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 670 op/s 2023-11-17T00:05:04.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:04 smithi023 ceph-mon[97824]: pgmap v488: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 704 op/s 2023-11-17T00:05:04.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:04 smithi173 ceph-mon[131534]: pgmap v488: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 704 op/s 2023-11-17T00:05:04.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:04 smithi156 ceph-mon[131105]: pgmap v488: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 704 op/s 2023-11-17T00:05:05.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 -- 172.21.15.23:0/3362703491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84107f30 msgr2=0x7fec8410a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 --2- 172.21.15.23:0/3362703491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84107f30 0x7fec8410a3b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00b0a0 tx=0x7fec7c00b3b0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 -- 172.21.15.23:0/3362703491 shutdown_connections 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 --2- 172.21.15.23:0/3362703491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84107f30 0x7fec8410a3b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 --2- 172.21.15.23:0/3362703491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84074b30 0x7fec84072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 --2- 172.21.15.23:0/3362703491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec840739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 -- 172.21.15.23:0/3362703491 >> 172.21.15.23:0/3362703491 conn(0x7fec840fb670 msgr2=0x7fec840fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 -- 172.21.15.23:0/3362703491 shutdown_connections 2023-11-17T00:05:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.432+0000 7fec83fff700 1 -- 172.21.15.23:0/3362703491 wait complete. 2023-11-17T00:05:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.433+0000 7fec83fff700 1 Processor -- start 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.433+0000 7fec83fff700 1 -- start start 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec84198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 0x7fec84198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84107f30 0x7fec8419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8419fc90 con 0x7fec84073550 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8419fdd0 con 0x7fec84074b30 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec83fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8419ff60 con 0x7fec84107f30 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec84198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec827fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 0x7fec84198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec84198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35756/0 (socket says 172.21.15.23:35756) 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.434+0000 7fec82ffd700 1 -- 172.21.15.23:0/1066444794 learned_addr learned my addr 172.21.15.23:0/1066444794 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec837fe700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84107f30 0x7fec8419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 -- 172.21.15.23:0/1066444794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84107f30 msgr2=0x7fec8419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84107f30 0x7fec8419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 -- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 msgr2=0x7fec84198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec84198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 -- 172.21.15.23:0/1066444794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009d00 con 0x7fec84074b30 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec827fc700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 0x7fec84198650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e9c0 tx=0x7fec7800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec6bfff700 1 -- 172.21.15.23:0/1066444794 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78004220 con 0x7fec84074b30 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.435+0000 7fec6bfff700 1 -- 172.21.15.23:0/1066444794 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec78004380 con 0x7fec84074b30 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.436+0000 7fec6bfff700 1 -- 172.21.15.23:0/1066444794 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78003be0 con 0x7fec84074b30 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.436+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8410dc60 con 0x7fec84074b30 2023-11-17T00:05:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.436+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8410dad0 con 0x7fec84074b30 2023-11-17T00:05:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.439+0000 7fec6bfff700 1 -- 172.21.15.23:0/1066444794 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fec78003d40 con 0x7fec84074b30 2023-11-17T00:05:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.440+0000 7fec9d011b80 1 Processor -- start 2023-11-17T00:05:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 -- start start 2023-11-17T00:05:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 0x5611e0462a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 0x5611e0463820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0464200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0368000 con 0x5611e0463d60 2023-11-17T00:05:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e03ba840 con 0x5611e02be6b0 2023-11-17T00:05:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec9d011b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e03b3cd0 con 0x5611e0462fb0 2023-11-17T00:05:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 0x5611e0462a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 0x5611e0463820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 0x5611e0462a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:35954/0 (socket says 172.21.15.23:35954) 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec837fe700 1 -- 172.21.15.23:0/586067552 learned_addr learned my addr 172.21.15.23:0/586067552 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec6bfff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 0x7fec6c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.441+0000 7fec827fc700 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0464200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 -- 172.21.15.23:0/586067552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 msgr2=0x5611e0462a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 0x5611e0462a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 -- 172.21.15.23:0/586067552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 msgr2=0x5611e0464200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec6bfff700 1 -- 172.21.15.23:0/1066444794 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fec78014070 con 0x7fec84074b30 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0464200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 -- 172.21.15.23:0/586067552 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611e02355d0 con 0x5611e0462fb0 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec82ffd700 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 0x5611e0463820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec7400bd20 tx=0x7fec740106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee959ee287919bd6 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec827fc700 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0464200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:05:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec697fa700 1 -- 172.21.15.23:0/586067552 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74010e10 con 0x5611e0462fb0 2023-11-17T00:05:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec697fa700 1 -- 172.21.15.23:0/586067552 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec74015070 con 0x5611e0462fb0 2023-11-17T00:05:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.442+0000 7fec697fa700 1 -- 172.21.15.23:0/586067552 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740196b0 con 0x5611e0462fb0 2023-11-17T00:05:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec837fe700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 0x7fec6c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 -- 172.21.15.23:0/586067552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 msgr2=0x5611e0463820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 0x5611e0463820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec7400bd20 tx=0x7fec740106b0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec837fe700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 0x7fec6c0a3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5611e045dee0 tx=0x7fec7c0099e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 -- 172.21.15.23:0/586067552 shutdown_connections 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0464200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e02be6b0 0x5611e0462a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 --2- 172.21.15.23:0/586067552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e0462fb0 0x5611e0463820 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.443+0000 7fec9d011b80 1 -- 172.21.15.23:0/586067552 >> 172.21.15.23:0/586067552 conn(0x5611e04581a0 msgr2=0x5611e045d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.444+0000 7fec9d011b80 1 -- 172.21.15.23:0/586067552 shutdown_connections 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.444+0000 7fec9d011b80 1 -- 172.21.15.23:0/586067552 wait complete. 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.444+0000 7fec9d011b80 1 -- start start 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.444+0000 7fec9d011b80 1 Processor -- start 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0473440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0355880 con 0x5611e0463d60 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0355520 con 0x5611e0462fb0 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e02378c0 con 0x5611e02be6b0 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec9d011b80 10 client.0 _ll_register_callbacks cb 0x5611e0373720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.445+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0473440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37130/0 (socket says 172.21.15.23:37130) 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec82ffd700 1 -- 172.21.15.23:0/4178581727 learned_addr learned my addr 172.21.15.23:0/4178581727 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 -- 172.21.15.23:0/4178581727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 msgr2=0x5611e046bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 -- 172.21.15.23:0/4178581727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 msgr2=0x5611e0473440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0473440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 -- 172.21.15.23:0/4178581727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7400b950 con 0x5611e0462fb0 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec82ffd700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec837fe700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0473440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:05:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec827fc700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046c400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec780ca990 tx=0x7fec78065870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa55bd671e18a server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.446+0000 7fec66ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.447+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7809a390 con 0x5611e0462fb0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.447+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fec78099f30 con 0x5611e0462fb0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.447+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611e0455960 con 0x5611e0462fb0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.447+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec7809df60 con 0x5611e0462fb0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.447+0000 7fec9d011b80 10 client.25625 fetch_fsmap learned FSMap version 29 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.448+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5611e037a130 con 0x5611e0462fb0 2023-11-17T00:05:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.448+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec78069020 con 0x5611e0462fb0 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.448+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fec78099520 con 0x5611e0462fb0 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 10 client.25625 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 10 client.25625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 20 client.25625 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e0477cf0 0x5611e047a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 4 client.25625 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611e0376040 con 0x5611e0477cf0 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.449+0000 7fec82ffd700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e0477cf0 0x5611e047a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.450+0000 7fec82ffd700 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e0477cf0 0x5611e047a100 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d1d3d3a60ca0885 server_cookie=ea61a649b855704c in_seq=0 out_seq=0 2023-11-17T00:05:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.450+0000 7fec66ff5700 10 client.25625 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.451+0000 7fec66ff5700 1 -- 172.21.15.23:0/4178581727 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5611e0376040 con 0x5611e0477cf0 2023-11-17T00:05:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.451+0000 7fec66ff5700 10 client.25625 handle_command_reply: tid=0 2023-11-17T00:05:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.451+0000 7fec9d011b80 1 client.25625 shutdown 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 msgr2=0x5611e046c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046c400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec780ca990 tx=0x7fec78065870 comp rx=0 tx=0).stop 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 shutdown_connections 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e0477cf0 0x5611e047a100 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0473440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 --2- 172.21.15.23:0/4178581727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046c400 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec66ff5700 0 client.25625 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 >> 172.21.15.23:0/4178581727 conn(0x5611e04581a0 msgr2=0x5611e045ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 shutdown_connections 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 1 -- 172.21.15.23:0/4178581727 wait complete. 2023-11-17T00:05:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.452+0000 7fec9d011b80 20 client.25625 trim_cache size 0 max 16384 2023-11-17T00:05:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 Processor -- start 2023-11-17T00:05:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 -- start start 2023-11-17T00:05:05.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 0x5611e0473400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 0x5611e046a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0463640 con 0x5611e0462fb0 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0463380 con 0x5611e0463d60 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec9d011b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0475020 con 0x5611e02be6b0 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.471+0000 7fec827fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 0x5611e0473400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 0x5611e046a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37144/0 (socket says 172.21.15.23:37144) 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 -- 172.21.15.23:0/1892571772 learned_addr learned my addr 172.21.15.23:0/1892571772 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:05.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 0x5611e046a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:35970/0 (socket says 172.21.15.23:35970) 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 -- 172.21.15.23:0/1892571772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 msgr2=0x5611e046a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 0x5611e046a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 -- 172.21.15.23:0/1892571772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 msgr2=0x5611e0473400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 0x5611e0473400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 -- 172.21.15.23:0/1892571772 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611e0379e80 con 0x5611e02be6b0 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec827fc700 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 0x5611e0473400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.472+0000 7fec82ffd700 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec74005920 tx=0x7fec74011f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6cbbe2939374f41 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.473+0000 7fec69ffb700 1 -- 172.21.15.23:0/1892571772 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7402e070 con 0x5611e02be6b0 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.473+0000 7fec69ffb700 1 -- 172.21.15.23:0/1892571772 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec74020070 con 0x5611e02be6b0 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.473+0000 7fec69ffb700 1 -- 172.21.15.23:0/1892571772 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7402a490 con 0x5611e02be6b0 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- 172.21.15.23:0/1892571772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 msgr2=0x5611e046a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec74005920 tx=0x7fec74011f80 comp rx=0 tx=0).stop 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- 172.21.15.23:0/1892571772 shutdown_connections 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0463d60 0x5611e046a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046a0a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 --2- 172.21.15.23:0/1892571772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0462fb0 0x5611e0473400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- 172.21.15.23:0/1892571772 >> 172.21.15.23:0/1892571772 conn(0x5611e04581a0 msgr2=0x5611e047ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- 172.21.15.23:0/1892571772 shutdown_connections 2023-11-17T00:05:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- 172.21.15.23:0/1892571772 wait complete. 2023-11-17T00:05:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.474+0000 7fec9d011b80 1 -- start start 2023-11-17T00:05:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.475+0000 7fec9d011b80 1 Processor -- start 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0472470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:35984/0 (socket says 172.21.15.23:35984) 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37146/0 (socket says 172.21.15.23:37146) 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 -- 172.21.15.23:0/2775556040 learned_addr learned my addr 172.21.15.23:0/2775556040 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e04729b0 con 0x5611e0463d60 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0472b20 con 0x5611e0462fb0 2023-11-17T00:05:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e0472ce0 con 0x5611e02be6b0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec9d011b80 10 client.0 _ll_register_callbacks cb 0x7fec78099b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 -- 172.21.15.23:0/2775556040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 msgr2=0x5611e046ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 -- 172.21.15.23:0/2775556040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 msgr2=0x5611e0472470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0472470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.476+0000 7fec827fc700 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611e037a130 con 0x5611e0462fb0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec827fc700 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec780ca670 tx=0x7fec78099d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc2cb53fc5d83c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec66ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78099f30 con 0x5611e0462fb0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec74015070 con 0x5611e0462fb0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fec7809df60 con 0x5611e0462fb0 2023-11-17T00:05:05.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec780b81a0 con 0x5611e0462fb0 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec9d011b80 10 client.25631 fetch_fsmap learned FSMap version 29 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5611e02355d0 con 0x5611e0462fb0 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.477+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec74015070 con 0x5611e0462fb0 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fec78095d20 con 0x5611e0462fb0 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 10 client.25631 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 10 client.25631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 20 client.25631 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e04cc6b0 0x5611e04ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 4 client.25631 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5611e0382600 con 0x5611e04cc6b0 2023-11-17T00:05:05.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec82ffd700 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e04cc6b0 0x5611e04ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec82ffd700 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e04cc6b0 0x5611e04ceac0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7baf6db4860779a server_cookie=bc159dca6e010a27 in_seq=0 out_seq=0 2023-11-17T00:05:05.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:05.478+0000 7fec66ff5700 10 client.25631 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:06 smithi023 ceph-mon[97824]: pgmap v489: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 697 op/s 2023-11-17T00:05:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.591+0000 7fec66ff5700 1 -- 172.21.15.23:0/2775556040 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5611e0382600 con 0x5611e04cc6b0 2023-11-17T00:05:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.591+0000 7fec66ff5700 10 client.25631 handle_command_reply: tid=0 2023-11-17T00:05:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.591+0000 7fec9d011b80 1 client.25631 shutdown 2023-11-17T00:05:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 msgr2=0x5611e046f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec780ca670 tx=0x7fec78099d00 comp rx=0 tx=0).stop 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 shutdown_connections 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5611e04cc6b0 0x5611e04ceac0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec66ff5700 0 client.25631 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e0463d60 0x5611e0472470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611e02be6b0 0x5611e046ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 --2- 172.21.15.23:0/2775556040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5611e0462fb0 0x5611e046f450 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 >> 172.21.15.23:0/2775556040 conn(0x5611e04581a0 msgr2=0x5611e046a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.594+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 shutdown_connections 2023-11-17T00:05:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.595+0000 7fec9d011b80 1 -- 172.21.15.23:0/2775556040 wait complete. 2023-11-17T00:05:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.595+0000 7fec9d011b80 20 client.25631 trim_cache size 0 max 16384 2023-11-17T00:05:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:05:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:05:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:05:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:05:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.596+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 msgr2=0x7fec6c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:06.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.596+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 0x7fec6c0a3680 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5611e045dee0 tx=0x7fec7c0099e0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.596+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 msgr2=0x7fec84198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.596+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 0x7fec84198650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e9c0 tx=0x7fec7800ecd0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 shutdown_connections 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fec6c0a11d0 0x7fec6c0a3680 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec84107f30 0x7fec8419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec84073550 0x7fec84198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 --2- 172.21.15.23:0/1066444794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec84074b30 0x7fec84198650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 >> 172.21.15.23:0/1066444794 conn(0x7fec840fb670 msgr2=0x7fec84103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 shutdown_connections 2023-11-17T00:05:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:06.597+0000 7fec83fff700 1 -- 172.21.15.23:0/1066444794 wait complete. 2023-11-17T00:05:06.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:06 smithi156 ceph-mon[131105]: pgmap v489: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 697 op/s 2023-11-17T00:05:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:06 smithi173 ceph-mon[131534]: pgmap v489: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 697 op/s 2023-11-17T00:05:08.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:08 smithi023 ceph-mon[97824]: pgmap v490: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 551 op/s 2023-11-17T00:05:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:08 smithi173 ceph-mon[131534]: pgmap v490: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 551 op/s 2023-11-17T00:05:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:08 smithi156 ceph-mon[131105]: pgmap v490: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 551 op/s 2023-11-17T00:05:10.108 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: In function 'void C_GatherBase::sub_finish(ContextType*, int) [with ContextType = MDSContext; ContextInstanceType = C_MDSInternalNoop]' thread 7efbf4990700 time 2023-11-17T00:05:09.996868+0000 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: 322: FAILED ceph_assert(waitfor.count(sub)) 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efc0678edbf] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: clone() 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: *** Caught signal (Aborted) ** 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: in thread 7efbf4990700 thread_name:md_log_replay 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:09.997+0000 7efbf4990700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: In function 'void C_GatherBase::sub_finish(ContextType*, int) [with ContextType = MDSContext; ContextInstanceType = C_MDSInternalNoop]' thread 7efbf4990700 time 2023-11-17T00:05:09.996868+0000 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: 322: FAILED ceph_assert(waitfor.count(sub)) 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.110 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-mon[131534]: pgmap v491: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 595 op/s 2023-11-17T00:05:10.111 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.111 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efc0678edbf] 2023-11-17T00:05:10.111 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.112 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: clone() 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7efc0553fcf0] 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: gsignal() 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: abort() 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efc0678ee19] 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.113 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 11: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 13: clone() 2023-11-17T00:05:10.114 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:09.998+0000 7efbf4990700 -1 *** Caught signal (Aborted) ** 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: in thread 7efbf4990700 thread_name:md_log_replay 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7efc0553fcf0] 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: gsignal() 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: abort() 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efc0678ee19] 2023-11-17T00:05:10.115 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 11: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 13: clone() 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.116 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: -1> 2023-11-17T00:05:09.997+0000 7efbf4990700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: In function 'void C_GatherBase::sub_finish(ContextType*, int) [with ContextType = MDSContext; ContextInstanceType = C_MDSInternalNoop]' thread 7efbf4990700 time 2023-11-17T00:05:09.996868+0000 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: 322: FAILED ceph_assert(waitfor.count(sub)) 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efc0678edbf] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.117 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: clone() 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 0> 2023-11-17T00:05:09.998+0000 7efbf4990700 -1 *** Caught signal (Aborted) ** 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: in thread 7efbf4990700 thread_name:md_log_replay 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7efc0553fcf0] 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: gsignal() 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: abort() 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efc0678ee19] 2023-11-17T00:05:10.118 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 11: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 13: clone() 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: -1> 2023-11-17T00:05:09.997+0000 7efbf4990700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: In function 'void C_GatherBase::sub_finish(ContextType*, int) [with ContextType = MDSContext; ContextInstanceType = C_MDSInternalNoop]' thread 7efbf4990700 time 2023-11-17T00:05:09.996868+0000 2023-11-17T00:05:10.119 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3906-g1c956514/rpm/el8/BUILD/ceph-18.0.0-3906-g1c956514/src/include/Context.h: 322: FAILED ceph_assert(waitfor.count(sub)) 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7efc0678edbf] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.120 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: clone() 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 0> 2023-11-17T00:05:09.998+0000 7efbf4990700 -1 *** Caught signal (Aborted) ** 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: in thread 7efbf4990700 thread_name:md_log_replay 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: ceph version 18.0.0-3906-g1c956514 (1c9565146c8647d4e00eb6c7ae626dc192680e7b) reef (dev) 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7efc0553fcf0] 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2: gsignal() 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 3: abort() 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7efc0678ee19] 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7efc0678ef85] 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 6: (C_GatherBase::sub_finish(MDSContext*, int)+0x1dc) [0x561f275e19cc] 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 7: (C_GatherBase::C_GatherSub::finish(int)+0x16) [0x561f275e1d26] 2023-11-17T00:05:10.121 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 8: (C_GatherBase::C_GatherSub::complete(int)+0xd) [0x561f275cbe8d] 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 9: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x561f275a21bd] 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 10: (MDLog::_replay_thread()+0xe1b) [0x561f2791c66b] 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 11: (MDLog::ReplayThread::entry()+0x11) [0x561f275cb481] 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc055351ca] 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 13: clone() 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[157356]: 2023-11-17T00:05:10.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:09 smithi156 ceph-mon[131105]: pgmap v491: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 595 op/s 2023-11-17T00:05:10.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:09 smithi023 ceph-mon[97824]: pgmap v491: 129 pgs: 129 active+clean; 177 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 595 op/s 2023-11-17T00:05:12.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:12 smithi023 ceph-mon[97824]: pgmap v492: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 619 op/s 2023-11-17T00:05:12.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:12 smithi173 ceph-mon[131534]: pgmap v492: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 619 op/s 2023-11-17T00:05:12.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:12 smithi156 ceph-mon[131105]: pgmap v492: 129 pgs: 129 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 619 op/s 2023-11-17T00:05:13.226 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:12 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service: Main process exited, code=exited, status=134/n/a 2023-11-17T00:05:13.235 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:13 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service: Failed with result 'exit-code'. 2023-11-17T00:05:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:14 smithi023 ceph-mon[97824]: pgmap v493: 129 pgs: 129 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 619 op/s 2023-11-17T00:05:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:14 smithi156 ceph-mon[131105]: pgmap v493: 129 pgs: 129 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 619 op/s 2023-11-17T00:05:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:14 smithi173 ceph-mon[131534]: pgmap v493: 129 pgs: 129 active+clean; 174 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 619 op/s 2023-11-17T00:05:16.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:16 smithi023 ceph-mon[97824]: pgmap v494: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 662 op/s 2023-11-17T00:05:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:16 smithi173 ceph-mon[131534]: pgmap v494: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 662 op/s 2023-11-17T00:05:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:16 smithi156 ceph-mon[131105]: pgmap v494: 129 pgs: 129 active+clean; 188 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 662 op/s 2023-11-17T00:05:18.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:18 smithi023 ceph-mon[97824]: pgmap v495: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 625 op/s 2023-11-17T00:05:18.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:18 smithi173 ceph-mon[131534]: pgmap v495: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 625 op/s 2023-11-17T00:05:18.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:18 smithi156 ceph-mon[131105]: pgmap v495: 129 pgs: 129 active+clean; 190 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 625 op/s 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.827+0000 7fac9dbff700 1 -- 172.21.15.23:0/2249459510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98106d10 msgr2=0x7fac98107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.827+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2249459510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98106d10 0x7fac98107120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac88009990 tx=0x7fac88009ca0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 -- 172.21.15.23:0/2249459510 shutdown_connections 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2249459510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac98108400 0x7fac9810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2249459510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98106d10 0x7fac98107120 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac88009990 tx=0x7fac88009ca0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2249459510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac98107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 -- 172.21.15.23:0/2249459510 >> 172.21.15.23:0/2249459510 conn(0x7fac980758b0 msgr2=0x7fac98077d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.828+0000 7fac9dbff700 1 -- 172.21.15.23:0/2249459510 shutdown_connections 2023-11-17T00:05:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.829+0000 7fac9dbff700 1 -- 172.21.15.23:0/2249459510 wait complete. 2023-11-17T00:05:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.834+0000 7fac9dbff700 1 Processor -- start 2023-11-17T00:05:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.835+0000 7fac9dbff700 1 -- start start 2023-11-17T00:05:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.835+0000 7fac9dbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac9dbff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98108400 0x7fac9819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60314/0 (socket says 172.21.15.23:60314) 2023-11-17T00:05:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac9dbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac981a1f90 0x7fac981a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac9dbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac981a4940 con 0x7fac98107a70 2023-11-17T00:05:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac9dbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac981a4a80 con 0x7fac981a1f90 2023-11-17T00:05:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac9dbff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac981a4c10 con 0x7fac98108400 2023-11-17T00:05:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.837+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac981a1f90 0x7fac981a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 -- 172.21.15.23:0/2710507364 learned_addr learned my addr 172.21.15.23:0/2710507364 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98108400 0x7fac9819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 -- 172.21.15.23:0/2710507364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98108400 msgr2=0x7fac9819cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98108400 0x7fac9819cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 -- 172.21.15.23:0/2710507364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac981a1f90 msgr2=0x7fac981a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac981a1f90 0x7fac981a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.838+0000 7fac977fe700 1 -- 172.21.15.23:0/2710507364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880095f0 con 0x7fac98107a70 2023-11-17T00:05:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.839+0000 7fac977fe700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac880098e0 tx=0x7fac88003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.849+0000 7fac94ff9700 1 -- 172.21.15.23:0/2710507364 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8801d070 con 0x7fac98107a70 2023-11-17T00:05:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.849+0000 7fac94ff9700 1 -- 172.21.15.23:0/2710507364 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac88003e70 con 0x7fac98107a70 2023-11-17T00:05:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.849+0000 7fac94ff9700 1 -- 172.21.15.23:0/2710507364 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8801d070 con 0x7fac98107a70 2023-11-17T00:05:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.849+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac98112340 con 0x7fac98107a70 2023-11-17T00:05:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.850+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac981121b0 con 0x7fac98107a70 2023-11-17T00:05:19.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.860+0000 7fac94ff9700 1 -- 172.21.15.23:0/2710507364 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fac8800fbb0 con 0x7fac98107a70 2023-11-17T00:05:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.862+0000 7facb150fb80 1 Processor -- start 2023-11-17T00:05:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.862+0000 7facb150fb80 1 -- start start 2023-11-17T00:05:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.863+0000 7facb150fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.863+0000 7facb150fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f758c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7facb150fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f758cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7facb150fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f7491000 con 0x5626f758cb10 2023-11-17T00:05:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7facb150fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f74e3840 con 0x5626f73e76b0 2023-11-17T00:05:19.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7facb150fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f74dccd0 con 0x5626f758bdb0 2023-11-17T00:05:19.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:49594/0 (socket says 172.21.15.23:49594) 2023-11-17T00:05:19.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.864+0000 7fac97fff700 1 -- 172.21.15.23:0/2802677382 learned_addr learned my addr 172.21.15.23:0/2802677382 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:19.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.865+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f758cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.865+0000 7fac97fff700 1 -- 172.21.15.23:0/2802677382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 msgr2=0x5626f758c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.865+0000 7fac977fe700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f758c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac97fff700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f758c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac97fff700 1 -- 172.21.15.23:0/2802677382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 msgr2=0x5626f758cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac97fff700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f758cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac97fff700 1 -- 172.21.15.23:0/2802677382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f735e5d0 con 0x5626f73e76b0 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac977fe700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f758c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f758cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.866+0000 7fac97fff700 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac90009f60 tx=0x7fac90009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257cc749300c9fda server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.867+0000 7fac94ff9700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 0x7fac800a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.867+0000 7fac7dffb700 1 -- 172.21.15.23:0/2802677382 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90011070 con 0x5626f73e76b0 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.868+0000 7fac7dffb700 1 -- 172.21.15.23:0/2802677382 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac90009a90 con 0x5626f73e76b0 2023-11-17T00:05:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.868+0000 7fac7dffb700 1 -- 172.21.15.23:0/2802677382 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90015d10 con 0x5626f73e76b0 2023-11-17T00:05:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.869+0000 7fac977fe700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 0x7fac800a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.869+0000 7facb150fb80 1 -- 172.21.15.23:0/2802677382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 msgr2=0x5626f758b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.869+0000 7facb150fb80 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac90009f60 tx=0x7fac90009f90 comp rx=0 tx=0).stop 2023-11-17T00:05:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 -- 172.21.15.23:0/2802677382 shutdown_connections 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f758cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f758b870 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 --2- 172.21.15.23:0/2802677382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f758c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 -- 172.21.15.23:0/2802677382 >> 172.21.15.23:0/2802677382 conn(0x5626f7580f50 msgr2=0x5626f7586100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7fac94ff9700 1 -- 172.21.15.23:0/2710507364 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fac880cdb30 con 0x7fac98107a70 2023-11-17T00:05:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.870+0000 7facb150fb80 1 -- 172.21.15.23:0/2802677382 shutdown_connections 2023-11-17T00:05:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.871+0000 7facb150fb80 1 -- 172.21.15.23:0/2802677382 wait complete. 2023-11-17T00:05:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.872+0000 7fac977fe700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 0x7fac800a3790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac8809a920 tx=0x7fac88027000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.872+0000 7facb150fb80 1 -- start start 2023-11-17T00:05:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.872+0000 7facb150fb80 1 Processor -- start 2023-11-17T00:05:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7594cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f7595210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f747e880 con 0x5626f758cb10 2023-11-17T00:05:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f747e520 con 0x5626f73e76b0 2023-11-17T00:05:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f73608c0 con 0x5626f758bdb0 2023-11-17T00:05:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.873+0000 7facb150fb80 10 client.0 _ll_register_callbacks cb 0x5626f749c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:19.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.875+0000 7fac97fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.875+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7594cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.875+0000 7fac96ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f7595210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.875+0000 7fac97fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60334/0 (socket says 172.21.15.23:60334) 2023-11-17T00:05:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.875+0000 7fac97fff700 1 -- 172.21.15.23:0/3653321275 learned_addr learned my addr 172.21.15.23:0/3653321275 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:19.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.876+0000 7fac97fff700 1 -- 172.21.15.23:0/3653321275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 msgr2=0x5626f7595210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.876+0000 7fac97fff700 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f7595210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.876+0000 7fac97fff700 1 -- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 msgr2=0x5626f7594cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.876+0000 7fac97fff700 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7594cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.876+0000 7fac97fff700 1 -- 172.21.15.23:0/3653321275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac90009c10 con 0x5626f758cb10 2023-11-17T00:05:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.877+0000 7fac97fff700 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac90015fd0 tx=0x7fac90004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f3ccfaf85f1365 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.877+0000 7fac76ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:05:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.878+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90011070 con 0x5626f758cb10 2023-11-17T00:05:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.879+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fac90024630 con 0x5626f758cb10 2023-11-17T00:05:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.879+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626f749f040 con 0x5626f758cb10 2023-11-17T00:05:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.880+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac900247f0 con 0x5626f758cb10 2023-11-17T00:05:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.880+0000 7facb150fb80 10 client.15603 fetch_fsmap learned FSMap version 29 2023-11-17T00:05:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.880+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5626f74a3130 con 0x5626f758cb10 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.881+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626f749f040 con 0x5626f758cb10 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fac90023410 con 0x5626f758cb10 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 10 client.15603 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 20 client.15603 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75a0940 0x5626f75a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 4 client.15603 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626f74ab600 con 0x5626f75a0940 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.882+0000 7fac96ffd700 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75a0940 0x5626f75a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.883+0000 7fac96ffd700 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75a0940 0x5626f75a2d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ce29d84104fde0 server_cookie=81151d459e17568a in_seq=0 out_seq=0 2023-11-17T00:05:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.883+0000 7fac76ffd700 10 client.15603 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.887+0000 7fac76ffd700 1 -- 172.21.15.23:0/3653321275 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5626f74ab600 con 0x5626f75a0940 2023-11-17T00:05:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.887+0000 7fac76ffd700 10 client.15603 handle_command_reply: tid=0 2023-11-17T00:05:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.887+0000 7facb150fb80 1 client.15603 shutdown 2023-11-17T00:05:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.890+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 msgr2=0x5626f759c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.890+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac90015fd0 tx=0x7fac90004af0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 shutdown_connections 2023-11-17T00:05:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75a0940 0x5626f75a2d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758cb10 0x5626f759c250 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7594cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 --2- 172.21.15.23:0/3653321275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758bdb0 0x5626f7595210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 >> 172.21.15.23:0/3653321275 conn(0x5626f7580f50 msgr2=0x5626f75857e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7fac76ffd700 0 client.15603 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.892+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 shutdown_connections 2023-11-17T00:05:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.893+0000 7facb150fb80 1 -- 172.21.15.23:0/3653321275 wait complete. 2023-11-17T00:05:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.893+0000 7facb150fb80 20 client.15603 trim_cache size 0 max 16384 2023-11-17T00:05:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.928+0000 7facb150fb80 1 Processor -- start 2023-11-17T00:05:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.928+0000 7facb150fb80 1 -- start start 2023-11-17T00:05:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7592eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f75933f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f7595d20 con 0x5626f758bdb0 2023-11-17T00:05:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f7594180 con 0x5626f73e76b0 2023-11-17T00:05:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7facb150fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f758c420 con 0x5626f758cb10 2023-11-17T00:05:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.929+0000 7fac96ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60346/0 (socket says 172.21.15.23:60346) 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 -- 172.21.15.23:0/1694595100 learned_addr learned my addr 172.21.15.23:0/1694595100 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac97fff700 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f75933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 -- 172.21.15.23:0/1694595100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 msgr2=0x5626f75933f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f75933f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 -- 172.21.15.23:0/1694595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 msgr2=0x5626f7592eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7592eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.930+0000 7fac96ffd700 1 -- 172.21.15.23:0/1694595100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f74a2e80 con 0x5626f758bdb0 2023-11-17T00:05:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.931+0000 7fac97fff700 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f75933f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:05:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.931+0000 7fac96ffd700 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x5626f75a3660 tx=0x7fac8c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f562928a622d45 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.931+0000 7fac7e7fc700 1 -- 172.21.15.23:0/1694595100 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c01c070 con 0x5626f758bdb0 2023-11-17T00:05:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.932+0000 7fac7e7fc700 1 -- 172.21.15.23:0/1694595100 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac8c016430 con 0x5626f758bdb0 2023-11-17T00:05:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.932+0000 7fac7e7fc700 1 -- 172.21.15.23:0/1694595100 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c024630 con 0x5626f758bdb0 2023-11-17T00:05:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.933+0000 7facb150fb80 1 -- 172.21.15.23:0/1694595100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 msgr2=0x5626f759c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.933+0000 7facb150fb80 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x5626f75a3660 tx=0x7fac8c008040 comp rx=0 tx=0).stop 2023-11-17T00:05:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.934+0000 7facb150fb80 1 -- 172.21.15.23:0/1694595100 shutdown_connections 2023-11-17T00:05:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.934+0000 7facb150fb80 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f75933f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.934+0000 7facb150fb80 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f7592eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.934+0000 7facb150fb80 1 --2- 172.21.15.23:0/1694595100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f759c210 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.934+0000 7facb150fb80 1 -- 172.21.15.23:0/1694595100 >> 172.21.15.23:0/1694595100 conn(0x5626f7580f50 msgr2=0x5626f75a76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.936+0000 7facb150fb80 1 -- 172.21.15.23:0/1694595100 shutdown_connections 2023-11-17T00:05:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.938+0000 7facb150fb80 1 -- 172.21.15.23:0/1694595100 wait complete. 2023-11-17T00:05:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.938+0000 7facb150fb80 1 -- start start 2023-11-17T00:05:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.939+0000 7facb150fb80 1 Processor -- start 2023-11-17T00:05:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.942+0000 7facb150fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f75983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.943+0000 7facb150fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f75988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.944+0000 7facb150fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f759b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.944+0000 7facb150fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f759b7e0 con 0x5626f758bdb0 2023-11-17T00:05:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.943+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f75983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.944+0000 7fac97fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f75983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:49616/0 (socket says 172.21.15.23:49616) 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.944+0000 7fac97fff700 1 -- 172.21.15.23:0/2655121502 learned_addr learned my addr 172.21.15.23:0/2655121502 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.945+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f759b950 con 0x5626f73e76b0 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.945+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f759bac0 con 0x5626f758cb10 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.945+0000 7facb150fb80 10 client.0 _ll_register_callbacks cb 0x7fac90018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.945+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f75988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.946+0000 7fac96ffd700 1 -- 172.21.15.23:0/2655121502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 msgr2=0x5626f759b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.946+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f759b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.946+0000 7fac96ffd700 1 -- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 msgr2=0x5626f75983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.946+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f75983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.946+0000 7fac96ffd700 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8c009800 con 0x5626f758bdb0 2023-11-17T00:05:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.947+0000 7fac96ffd700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f75988e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac8c008010 tx=0x7fac8c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba39700238ee746f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.947+0000 7fac76ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:05:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.947+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c01c050 con 0x5626f758bdb0 2023-11-17T00:05:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.948+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fac8c016860 con 0x5626f758bdb0 2023-11-17T00:05:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.948+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626f757e710 con 0x5626f758bdb0 2023-11-17T00:05:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.949+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac8c01a040 con 0x5626f758bdb0 2023-11-17T00:05:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.949+0000 7facb150fb80 10 client.15609 fetch_fsmap learned FSMap version 29 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.949+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5626f72f3c00 con 0x5626f758bdb0 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626f757e710 con 0x5626f758bdb0 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fac8c02b070 con 0x5626f758bdb0 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7facb150fb80 10 client.15609 fetch_fsmap finished waiting for FSMap version 29 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7facb150fb80 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7facb150fb80 20 client.15609 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.950+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75f54c0 0x5626f75f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.951+0000 7facb150fb80 4 client.15609 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:05:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.951+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5626f732bf10 con 0x5626f75f54c0 2023-11-17T00:05:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.951+0000 7fac97fff700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75f54c0 0x5626f75f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.952+0000 7fac97fff700 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75f54c0 0x5626f75f78d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686faff457546be6 server_cookie=cb4aef511c23016e in_seq=0 out_seq=0 2023-11-17T00:05:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:19.952+0000 7fac76ffd700 10 client.15609 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:20.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:19 smithi173 ceph-mon[131534]: pgmap v496: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 670 op/s 2023-11-17T00:05:20.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:19 smithi156 ceph-mon[131105]: pgmap v496: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 670 op/s 2023-11-17T00:05:20.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:19 smithi023 ceph-mon[97824]: pgmap v496: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 670 op/s 2023-11-17T00:05:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.566+0000 7fac76ffd700 1 -- 172.21.15.23:0/2655121502 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5626f732bf10 con 0x5626f75f54c0 2023-11-17T00:05:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.566+0000 7fac76ffd700 10 client.15609 handle_command_reply: tid=0 2023-11-17T00:05:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.566+0000 7facb150fb80 1 client.15609 shutdown 2023-11-17T00:05:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.567+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 msgr2=0x5626f75988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.567+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f75988e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac8c008010 tx=0x7fac8c0095e0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 shutdown_connections 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626f758cb10 0x5626f759b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626f73e76b0 0x5626f75983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626f75f54c0 0x5626f75f78d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 --2- 172.21.15.23:0/2655121502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626f758bdb0 0x5626f75988e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 >> 172.21.15.23:0/2655121502 conn(0x5626f7580f50 msgr2=0x5626f7592eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.568+0000 7fac76ffd700 0 client.15609 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.569+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 shutdown_connections 2023-11-17T00:05:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.569+0000 7facb150fb80 1 -- 172.21.15.23:0/2655121502 wait complete. 2023-11-17T00:05:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.570+0000 7facb150fb80 20 client.15609 trim_cache size 0 max 16384 2023-11-17T00:05:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:05:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:05:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:05:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:05:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.571+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 msgr2=0x7fac800a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 0x7fac800a3790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac8809a920 tx=0x7fac88027000 comp rx=0 tx=0).stop 2023-11-17T00:05:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 msgr2=0x7fac9819c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac880098e0 tx=0x7fac88003cd0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 shutdown_connections 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac981a1f90 0x7fac981a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac98107a70 0x7fac9819c9f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fac800a12e0 0x7fac800a3790 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 --2- 172.21.15.23:0/2710507364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac98108400 0x7fac9819cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.572+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 >> 172.21.15.23:0/2710507364 conn(0x7fac980758b0 msgr2=0x7fac98077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.573+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 shutdown_connections 2023-11-17T00:05:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:20.573+0000 7fac9dbff700 1 -- 172.21.15.23:0/2710507364 wait complete. 2023-11-17T00:05:22.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:22 smithi023 ceph-mon[97824]: pgmap v497: 129 pgs: 129 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 607 op/s 2023-11-17T00:05:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:22 smithi173 ceph-mon[131534]: pgmap v497: 129 pgs: 129 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 607 op/s 2023-11-17T00:05:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:22 smithi156 ceph-mon[131105]: pgmap v497: 129 pgs: 129 active+clean; 199 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 607 op/s 2023-11-17T00:05:23.748 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:23 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service: Service RestartSec=10s expired, scheduling restart. 2023-11-17T00:05:23.748 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:23 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service: Scheduled restart job, restart counter is at 1. 2023-11-17T00:05:23.748 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:23 smithi173 systemd[1]: Stopped Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:05:23.748 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:23 smithi173 systemd[1]: Starting Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:05:23.749 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:23 smithi173 podman[160326]: 2023-11-17T00:05:24.044 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[160342]: starting mds.c at 2023-11-17T00:05:24.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:24 smithi173 ceph-mon[131534]: pgmap v498: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 625 op/s 2023-11-17T00:05:24.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:24 smithi173 ceph-mon[131534]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2023-11-17T00:05:24.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:24 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2023-11-17T00:05:24.357 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:24 smithi173 bash[160326]: 9e52830c2d9bcb80e8b515df83d595fe745e247a0c98bbaafb0df1de757d3d03 2023-11-17T00:05:24.357 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:05:24 smithi173 systemd[1]: Started Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:05:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:24 smithi023 ceph-mon[97824]: pgmap v498: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 625 op/s 2023-11-17T00:05:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:24 smithi023 ceph-mon[97824]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2023-11-17T00:05:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:24 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2023-11-17T00:05:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:24 smithi156 ceph-mon[131105]: pgmap v498: 129 pgs: 129 active+clean; 201 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 625 op/s 2023-11-17T00:05:24.996 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:24 smithi156 ceph-mon[131105]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2023-11-17T00:05:25.536 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:24 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 5 up:standby 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:25 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6836/2948132470,v1:172.21.15.173:6837/2948132470] up:boot 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:25 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:25 smithi156 ceph-mon[131105]: pgmap v499: 129 pgs: 129 active+clean; 199 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 730 op/s 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:25 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:05:26.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:25 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6836/2948132470,v1:172.21.15.173:6837/2948132470] up:boot 2023-11-17T00:05:26.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:25 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:26.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-17T00:05:26.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:25 smithi173 ceph-mon[131534]: pgmap v499: 129 pgs: 129 active+clean; 199 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 730 op/s 2023-11-17T00:05:26.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:25 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:05:26.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:25 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6836/2948132470,v1:172.21.15.173:6837/2948132470] up:boot 2023-11-17T00:05:26.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:25 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:26.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-17T00:05:26.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:25 smithi023 ceph-mon[97824]: pgmap v499: 129 pgs: 129 active+clean; 199 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 730 op/s 2023-11-17T00:05:26.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:25 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:05:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: pgmap v500: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 667 op/s 2023-11-17T00:05:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:05:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:05:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:05:28.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:05:28.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:28 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:05:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: pgmap v500: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 667 op/s 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:28 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: pgmap v500: 129 pgs: 129 active+clean; 202 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 667 op/s 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:05:28.692 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:28 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:05:30.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:29 smithi173 ceph-mon[131534]: pgmap v501: 129 pgs: 129 active+clean; 215 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 712 op/s 2023-11-17T00:05:30.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:29 smithi156 ceph-mon[131105]: pgmap v501: 129 pgs: 129 active+clean; 215 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 712 op/s 2023-11-17T00:05:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:29 smithi023 ceph-mon[97824]: pgmap v501: 129 pgs: 129 active+clean; 215 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 712 op/s 2023-11-17T00:05:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:32 smithi023 ceph-mon[97824]: pgmap v502: 129 pgs: 129 active+clean; 217 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 676 op/s 2023-11-17T00:05:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:32 smithi156 ceph-mon[131105]: pgmap v502: 129 pgs: 129 active+clean; 217 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 676 op/s 2023-11-17T00:05:32.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:32 smithi173 ceph-mon[131534]: pgmap v502: 129 pgs: 129 active+clean; 217 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 676 op/s 2023-11-17T00:05:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 -- 172.21.15.23:0/1922916666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f4071620 msgr2=0x7f14f4071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/1922916666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f4071620 0x7f14f4071a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009990 tx=0x7f14e8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 -- 172.21.15.23:0/1922916666 shutdown_connections 2023-11-17T00:05:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/1922916666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f410c740 0x7f14f410ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/1922916666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f4071620 0x7f14f4071a30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/1922916666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4109da0 0x7f14f410c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.792+0000 7f14fa3ff700 1 -- 172.21.15.23:0/1922916666 >> 172.21.15.23:0/1922916666 conn(0x7f14f406d1b0 msgr2=0x7f14f406f620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.794+0000 7f14fa3ff700 1 -- 172.21.15.23:0/1922916666 shutdown_connections 2023-11-17T00:05:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.794+0000 7f14fa3ff700 1 -- 172.21.15.23:0/1922916666 wait complete. 2023-11-17T00:05:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.797+0000 7f14fa3ff700 1 Processor -- start 2023-11-17T00:05:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.798+0000 7f14fa3ff700 1 -- start start 2023-11-17T00:05:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4071620 0x7f14f41a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f410c740 0x7f14f41ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41ac970 con 0x7f14f4071620 2023-11-17T00:05:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41acab0 con 0x7f14f4109da0 2023-11-17T00:05:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14fa3ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f41acc40 con 0x7f14f410c740 2023-11-17T00:05:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14f37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14f37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42312/0 (socket says 172.21.15.23:42312) 2023-11-17T00:05:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.799+0000 7f14f37fe700 1 -- 172.21.15.23:0/4138480912 learned_addr learned my addr 172.21.15.23:0/4138480912 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f899c700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f410c740 0x7f14f41ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f3fff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4071620 0x7f14f41a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f37fe700 1 -- 172.21.15.23:0/4138480912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f410c740 msgr2=0x7f14f41ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f37fe700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f410c740 0x7f14f41ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f37fe700 1 -- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4071620 msgr2=0x7f14f41a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f37fe700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4071620 0x7f14f41a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.800+0000 7f14f37fe700 1 -- 172.21.15.23:0/4138480912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4009520 con 0x7f14f4109da0 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14f37fe700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14e400eb30 tx=0x7f14e400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14f17fa700 1 -- 172.21.15.23:0/4138480912 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4009880 con 0x7f14f4109da0 2023-11-17T00:05:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e80095f0 con 0x7f14f4109da0 2023-11-17T00:05:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14f17fa700 1 -- 172.21.15.23:0/4138480912 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14e40044e0 con 0x7f14f4109da0 2023-11-17T00:05:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14f17fa700 1 -- 172.21.15.23:0/4138480912 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4010430 con 0x7f14f4109da0 2023-11-17T00:05:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.801+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f411aa10 con 0x7f14f4109da0 2023-11-17T00:05:33.811 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.809+0000 7f14f17fa700 1 -- 172.21.15.23:0/4138480912 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f14e4004020 con 0x7f14f4109da0 2023-11-17T00:05:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.814+0000 7f14f17fa700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 0x7f14dc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 Processor -- start 2023-11-17T00:05:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 -- start start 2023-11-17T00:05:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f14f17fa700 1 -- 172.21.15.23:0/4138480912 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f14e4014070 con 0x7f14f4109da0 2023-11-17T00:05:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7aca36b0 0x559f7ae47870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ad4d000 con 0x559f7ae48b10 2023-11-17T00:05:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ad9f840 con 0x559f7ae47db0 2023-11-17T00:05:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.815+0000 7f150dc61b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ad98cd0 con 0x559f7aca36b0 2023-11-17T00:05:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.816+0000 7f14f3fff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 0x7f14dc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.816+0000 7f14f37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.816+0000 7f14f899c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.816+0000 7f14f899c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60264/0 (socket says 172.21.15.23:60264) 2023-11-17T00:05:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.816+0000 7f14f899c700 1 -- 172.21.15.23:0/424793610 learned_addr learned my addr 172.21.15.23:0/424793610 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.817+0000 7f14f899c700 1 -- 172.21.15.23:0/424793610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7aca36b0 msgr2=0x559f7ae47870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-17T00:05:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.817+0000 7f14f899c700 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7aca36b0 0x559f7ae47870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.817+0000 7f14f899c700 1 -- 172.21.15.23:0/424793610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 msgr2=0x559f7ae485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.817+0000 7f14f899c700 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.817+0000 7f14f899c700 1 -- 172.21.15.23:0/424793610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f7ac1a5d0 con 0x559f7ae48b10 2023-11-17T00:05:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.819+0000 7f14f899c700 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009dc0 tx=0x7f14ec0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df419a831b21b97 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.820+0000 7f14da7fc700 1 -- 172.21.15.23:0/424793610 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec014610 con 0x559f7ae48b10 2023-11-17T00:05:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.820+0000 7f14da7fc700 1 -- 172.21.15.23:0/424793610 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14ec00f050 con 0x559f7ae48b10 2023-11-17T00:05:33.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.820+0000 7f14da7fc700 1 -- 172.21.15.23:0/424793610 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec014610 con 0x559f7ae48b10 2023-11-17T00:05:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.821+0000 7f150dc61b80 1 -- 172.21.15.23:0/424793610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 msgr2=0x559f7ae48f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.821+0000 7f150dc61b80 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009dc0 tx=0x7f14ec0077e0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.821+0000 7f14f3fff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 0x7f14dc0a3750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006010 tx=0x7f14e800ba70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 -- 172.21.15.23:0/424793610 shutdown_connections 2023-11-17T00:05:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7ae48b10 0x559f7ae48f80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7aca36b0 0x559f7ae47870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 --2- 172.21.15.23:0/424793610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 -- 172.21.15.23:0/424793610 >> 172.21.15.23:0/424793610 conn(0x559f7ae3cf50 msgr2=0x559f7ae42100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.822+0000 7f150dc61b80 1 -- 172.21.15.23:0/424793610 shutdown_connections 2023-11-17T00:05:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.823+0000 7f150dc61b80 1 -- 172.21.15.23:0/424793610 wait complete. 2023-11-17T00:05:33.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.825+0000 7f150dc61b80 1 -- start start 2023-11-17T00:05:33.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.826+0000 7f150dc61b80 1 Processor -- start 2023-11-17T00:05:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae50c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ad3a880 con 0x559f7aca36b0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ad3a520 con 0x559f7ae47db0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ac1c8c0 con 0x559f7ae48b10 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.828+0000 7f150dc61b80 10 client.0 _ll_register_callbacks cb 0x559f7ad58720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.829+0000 7f14f899c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.829+0000 7f14f899c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:41448/0 (socket says 172.21.15.23:41448) 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.829+0000 7f14f899c700 1 -- 172.21.15.23:0/450118615 learned_addr learned my addr 172.21.15.23:0/450118615 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f3fff700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae50c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 -- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 msgr2=0x559f7ae511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f37fe700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 -- 172.21.15.23:0/450118615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 msgr2=0x559f7ae50c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae50c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 -- 172.21.15.23:0/450118615 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec009a20 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f3fff700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae50c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14f899c700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009490 tx=0x7f14ec014c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c52369701b3cdd2 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.830+0000 7f14d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.831+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec004cd0 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.831+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f14ec004e30 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.831+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f7ae3a710 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.832+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14ec006080 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 10 client.25690 fetch_fsmap learned FSMap version 31 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559f7ad5f130 con 0x559f7ae48b10 2023-11-17T00:05:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559f7ae3a710 con 0x559f7ae48b10 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f14ec0055d0 con 0x559f7ae48b10 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 10 client.25690 fetch_fsmap finished waiting for FSMap version 31 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 20 client.25690 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.833+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7ae5c960 0x559f7ae5ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.834+0000 7f150dc61b80 4 client.25690 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:05:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.834+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f7ad5b040 con 0x559f7ae5c960 2023-11-17T00:05:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.835+0000 7f14f3fff700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7ae5c960 0x559f7ae5ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.839+0000 7f14f3fff700 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7ae5c960 0x559f7ae5ed70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9e6dd68e03f544 server_cookie=db6893086f7b4b81 in_seq=0 out_seq=0 2023-11-17T00:05:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.839+0000 7f14d37fe700 10 client.25690 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.839+0000 7f14d37fe700 1 -- 172.21.15.23:0/450118615 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559f7ad5b040 con 0x559f7ae5c960 2023-11-17T00:05:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.839+0000 7f14d37fe700 10 client.25690 handle_command_reply: tid=0 2023-11-17T00:05:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.840+0000 7f150dc61b80 1 client.25690 shutdown 2023-11-17T00:05:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.841+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 msgr2=0x559f7ae581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.841+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009490 tx=0x7f14ec014c70 comp rx=0 tx=0).stop 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 shutdown_connections 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae48b10 0x559f7ae581e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7ae5c960 0x559f7ae5ed70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae50c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 --2- 172.21.15.23:0/450118615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae47db0 0x559f7ae511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 >> 172.21.15.23:0/450118615 conn(0x559f7ae3cf50 msgr2=0x559f7ae417e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f14d37fe700 0 client.25690 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 shutdown_connections 2023-11-17T00:05:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.842+0000 7f150dc61b80 1 -- 172.21.15.23:0/450118615 wait complete. 2023-11-17T00:05:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.844+0000 7f150dc61b80 20 client.25690 trim_cache size 0 max 16384 2023-11-17T00:05:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.879+0000 7f150dc61b80 1 Processor -- start 2023-11-17T00:05:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.879+0000 7f150dc61b80 1 -- start start 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.880+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae48ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.880+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae58100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae4fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f150dc61b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae4ed20 con 0x559f7aca36b0 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f150dc61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae4c630 con 0x559f7ae48b10 2023-11-17T00:05:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f150dc61b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae518c0 con 0x559f7ae47db0 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f14f37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae58100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f14f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae48ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f14f37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae58100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:41458/0 (socket says 172.21.15.23:41458) 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f14f37fe700 1 -- 172.21.15.23:0/486269028 learned_addr learned my addr 172.21.15.23:0/486269028 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.881+0000 7f14f899c700 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae4fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.882+0000 7f14f3fff700 1 -- 172.21.15.23:0/486269028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 msgr2=0x559f7ae58100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.882+0000 7f14f3fff700 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae58100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.882+0000 7f14f3fff700 1 -- 172.21.15.23:0/486269028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 msgr2=0x559f7ae4fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.882+0000 7f14f3fff700 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae4fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.882+0000 7f14f3fff700 1 -- 172.21.15.23:0/486269028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f7ad5ee80 con 0x559f7aca36b0 2023-11-17T00:05:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.883+0000 7f14f3fff700 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae48ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f14e80098e0 tx=0x7f14e8022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428e294ffc3d2bfe server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.884+0000 7f14daffd700 1 -- 172.21.15.23:0/486269028 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8031440 con 0x559f7aca36b0 2023-11-17T00:05:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.884+0000 7f14daffd700 1 -- 172.21.15.23:0/486269028 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14e801e040 con 0x559f7aca36b0 2023-11-17T00:05:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.884+0000 7f14daffd700 1 -- 172.21.15.23:0/486269028 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8004a30 con 0x559f7aca36b0 2023-11-17T00:05:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.886+0000 7f150dc61b80 1 -- 172.21.15.23:0/486269028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 msgr2=0x559f7ae48ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.886+0000 7f150dc61b80 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae48ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f14e80098e0 tx=0x7f14e8022000 comp rx=0 tx=0).stop 2023-11-17T00:05:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.887+0000 7f150dc61b80 1 -- 172.21.15.23:0/486269028 shutdown_connections 2023-11-17T00:05:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.887+0000 7f150dc61b80 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae4fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.887+0000 7f150dc61b80 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae48ee0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.887+0000 7f150dc61b80 1 --2- 172.21.15.23:0/486269028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae58100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.887+0000 7f150dc61b80 1 -- 172.21.15.23:0/486269028 >> 172.21.15.23:0/486269028 conn(0x559f7ae3cf50 msgr2=0x559f7ae63690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.889+0000 7f150dc61b80 1 -- 172.21.15.23:0/486269028 shutdown_connections 2023-11-17T00:05:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.889+0000 7f150dc61b80 1 -- 172.21.15.23:0/486269028 wait complete. 2023-11-17T00:05:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.889+0000 7f150dc61b80 1 -- start start 2023-11-17T00:05:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 Processor -- start 2023-11-17T00:05:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae54330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae54870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae577e0 con 0x559f7aca36b0 2023-11-17T00:05:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae57950 con 0x559f7ae48b10 2023-11-17T00:05:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f7ae57b10 con 0x559f7ae47db0 2023-11-17T00:05:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.890+0000 7f150dc61b80 10 client.0 _ll_register_callbacks cb 0x7f14e4095d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.891+0000 7f14f899c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.891+0000 7f14f899c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42358/0 (socket says 172.21.15.23:42358) 2023-11-17T00:05:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.891+0000 7f14f899c700 1 -- 172.21.15.23:0/2189042645 learned_addr learned my addr 172.21.15.23:0/2189042645 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.892+0000 7f14f3fff700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae54330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.892+0000 7f14f37fe700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae54870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f899c700 1 -- 172.21.15.23:0/2189042645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 msgr2=0x559f7ae54870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f899c700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae54870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f899c700 1 -- 172.21.15.23:0/2189042645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 msgr2=0x559f7ae54330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f899c700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae54330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f899c700 1 -- 172.21.15.23:0/2189042645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f4117d60 con 0x559f7ae48b10 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f3fff700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae54330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:05:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.893+0000 7f14f37fe700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae54870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:05:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.894+0000 7f14d37fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:05:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.894+0000 7f14f899c700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007820 tx=0x7f14ec02fd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=977d4da168f47c1e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.895+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec02f5c0 con 0x559f7ae48b10 2023-11-17T00:05:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.895+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f14ec0139b0 con 0x559f7ae48b10 2023-11-17T00:05:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.897+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14ec00f050 con 0x559f7ae48b10 2023-11-17T00:05:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.897+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14ec02f5c0 con 0x559f7ae48b10 2023-11-17T00:05:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.898+0000 7f150dc61b80 10 client.25664 fetch_fsmap learned FSMap version 31 2023-11-17T00:05:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.898+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559f7abafc00 con 0x559f7ae48b10 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14ec00f050 con 0x559f7ae48b10 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f14ec01d8d0 con 0x559f7ae48b10 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 10 client.25664 fetch_fsmap finished waiting for FSMap version 31 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 10 client.25664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 20 client.25664 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7aeb1420 0x559f7aeb3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 4 client.25664 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:05:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559f7ad67600 con 0x559f7aeb1420 2023-11-17T00:05:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.900+0000 7f14f3fff700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7aeb1420 0x559f7aeb3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.902+0000 7f14f3fff700 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7aeb1420 0x559f7aeb3830 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a2c3e9bd4d56099 server_cookie=92e17e82d8edfd13 in_seq=0 out_seq=0 2023-11-17T00:05:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:33.902+0000 7f14d37fe700 10 client.25664 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:34 smithi023 ceph-mon[97824]: pgmap v503: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 668 op/s 2023-11-17T00:05:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:34 smithi173 ceph-mon[131534]: pgmap v503: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 668 op/s 2023-11-17T00:05:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:34 smithi156 ceph-mon[131105]: pgmap v503: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 668 op/s 2023-11-17T00:05:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.438+0000 7f14d37fe700 1 -- 172.21.15.23:0/2189042645 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559f7ad67600 con 0x559f7aeb1420 2023-11-17T00:05:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.438+0000 7f14d37fe700 10 client.25664 handle_command_reply: tid=0 2023-11-17T00:05:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.438+0000 7f150dc61b80 1 client.25664 shutdown 2023-11-17T00:05:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.438+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 msgr2=0x559f7ae572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.438+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007820 tx=0x7f14ec02fd80 comp rx=0 tx=0).stop 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 shutdown_connections 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f7ae48b10 0x559f7ae572a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x559f7aeb1420 0x559f7aeb3830 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f14d37fe700 0 client.25664 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f7aca36b0 0x559f7ae54330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 --2- 172.21.15.23:0/2189042645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ae47db0 0x559f7ae54870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.439+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 >> 172.21.15.23:0/2189042645 conn(0x559f7ae3cf50 msgr2=0x559f7ae4fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:35.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.441+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 shutdown_connections 2023-11-17T00:05:35.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.441+0000 7f150dc61b80 1 -- 172.21.15.23:0/2189042645 wait complete. 2023-11-17T00:05:35.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.441+0000 7f150dc61b80 20 client.25664 trim_cache size 0 max 16384 2023-11-17T00:05:35.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:05:35.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:05:35.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:05:35.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:05:35.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 msgr2=0x7f14dc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 0x7f14dc0a3750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006010 tx=0x7f14e800ba70 comp rx=0 tx=0).stop 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 msgr2=0x7f14f41a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14e400eb30 tx=0x7f14e400eef0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 shutdown_connections 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f14dc0a12a0 0x7f14dc0a3750 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14f410c740 0x7f14f41ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f4071620 0x7f14f41a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 --2- 172.21.15.23:0/4138480912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14f4109da0 0x7f14f41a5330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.444+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 >> 172.21.15.23:0/4138480912 conn(0x7f14f406d1b0 msgr2=0x7f14f410b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.445+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 shutdown_connections 2023-11-17T00:05:35.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:35.445+0000 7f14fa3ff700 1 -- 172.21.15.23:0/4138480912 wait complete. 2023-11-17T00:05:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:36 smithi023 ceph-mon[97824]: pgmap v504: 129 pgs: 129 active+clean; 228 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 630 op/s 2023-11-17T00:05:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:36 smithi173 ceph-mon[131534]: pgmap v504: 129 pgs: 129 active+clean; 228 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 630 op/s 2023-11-17T00:05:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:36 smithi156 ceph-mon[131105]: pgmap v504: 129 pgs: 129 active+clean; 228 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.3 MiB/s wr, 630 op/s 2023-11-17T00:05:37.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[156678]: *** Caught signal (Segmentation fault) ** 2023-11-17T00:05:37.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[156678]: in thread 7ff7381cb700 thread_name:MR_Finisher 2023-11-17T00:05:38.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:38 smithi023 ceph-mon[97824]: pgmap v505: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 491 op/s 2023-11-17T00:05:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:38 smithi173 ceph-mon[131534]: pgmap v505: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 491 op/s 2023-11-17T00:05:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:38 smithi156 ceph-mon[131105]: pgmap v505: 129 pgs: 129 active+clean; 224 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 491 op/s 2023-11-17T00:05:40.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:39 smithi023 ceph-mon[97824]: pgmap v506: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 526 op/s 2023-11-17T00:05:40.352 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:40 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service: Main process exited, code=exited, status=139/n/a 2023-11-17T00:05:40.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:39 smithi173 ceph-mon[131534]: pgmap v506: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 526 op/s 2023-11-17T00:05:40.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:39 smithi156 ceph-mon[131105]: pgmap v506: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 526 op/s 2023-11-17T00:05:40.857 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:40 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service: Failed with result 'exit-code'. 2023-11-17T00:05:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:42 smithi023 ceph-mon[97824]: pgmap v507: 129 pgs: 129 active+clean; 227 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 482 op/s 2023-11-17T00:05:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:42 smithi173 ceph-mon[131534]: pgmap v507: 129 pgs: 129 active+clean; 227 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 482 op/s 2023-11-17T00:05:42.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:42 smithi156 ceph-mon[131105]: pgmap v507: 129 pgs: 129 active+clean; 227 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 482 op/s 2023-11-17T00:05:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:44 smithi023 ceph-mon[97824]: pgmap v508: 129 pgs: 129 active+clean; 238 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 541 op/s 2023-11-17T00:05:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:44 smithi173 ceph-mon[131534]: pgmap v508: 129 pgs: 129 active+clean; 238 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 541 op/s 2023-11-17T00:05:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:44 smithi156 ceph-mon[131105]: pgmap v508: 129 pgs: 129 active+clean; 238 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 541 op/s 2023-11-17T00:05:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:46 smithi173 ceph-mon[131534]: pgmap v509: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 598 op/s 2023-11-17T00:05:46.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:46 smithi156 ceph-mon[131105]: pgmap v509: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 598 op/s 2023-11-17T00:05:46.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:46 smithi023 ceph-mon[97824]: pgmap v509: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 598 op/s 2023-11-17T00:05:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:48 smithi156 ceph-mon[131105]: pgmap v510: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 546 op/s 2023-11-17T00:05:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:48 smithi173 ceph-mon[131534]: pgmap v510: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 546 op/s 2023-11-17T00:05:48.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:48 smithi023 ceph-mon[97824]: pgmap v510: 129 pgs: 129 active+clean; 241 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 546 op/s 2023-11-17T00:05:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 -- 172.21.15.23:0/1782316455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8100250 msgr2=0x7f7be8100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/1782316455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8100250 0x7f7be8100660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009990 tx=0x7f7bd0009ca0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 -- 172.21.15.23:0/1782316455 shutdown_connections 2023-11-17T00:05:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/1782316455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101dd0 0x7f7be8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/1782316455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8100250 0x7f7be8100660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/1782316455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8101440 0x7f7be8101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.800+0000 7f7bed7ff700 1 -- 172.21.15.23:0/1782316455 >> 172.21.15.23:0/1782316455 conn(0x7f7be80fb850 msgr2=0x7f7be80fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.801+0000 7f7bed7ff700 1 -- 172.21.15.23:0/1782316455 shutdown_connections 2023-11-17T00:05:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.801+0000 7f7bed7ff700 1 -- 172.21.15.23:0/1782316455 wait complete. 2023-11-17T00:05:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.802+0000 7f7bed7ff700 1 Processor -- start 2023-11-17T00:05:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.802+0000 7f7bed7ff700 1 -- start start 2023-11-17T00:05:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.802+0000 7f7bed7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8100250 0x7f7be8194060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.802+0000 7f7bed7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7bed7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8101dd0 0x7f7be81acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7bed7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8199690 con 0x7f7be8100250 2023-11-17T00:05:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7bed7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8199800 con 0x7f7be8101dd0 2023-11-17T00:05:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7bed7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be81ad030 con 0x7f7be8101440 2023-11-17T00:05:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7be6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7be77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8101dd0 0x7f7be81acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7be6ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37108/0 (socket says 172.21.15.23:37108) 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7be6ffd700 1 -- 172.21.15.23:0/419253579 learned_addr learned my addr 172.21.15.23:0/419253579 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.803+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8100250 0x7f7be8194060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 -- 172.21.15.23:0/419253579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8101dd0 msgr2=0x7f7be81acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8101dd0 0x7f7be81acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 -- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8100250 msgr2=0x7f7be8194060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8100250 0x7f7be8194060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 -- 172.21.15.23:0/419253579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd00095f0 con 0x7f7be8101440 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800b530 tx=0x7f7bd800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be4ff9700 1 -- 172.21.15.23:0/419253579 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8011830 con 0x7f7be8101440 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be810bd00 con 0x7f7be8101440 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be4ff9700 1 -- 172.21.15.23:0/419253579 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bd8011e50 con 0x7f7be8101440 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.804+0000 7f7be4ff9700 1 -- 172.21.15.23:0/419253579 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800f2e0 con 0x7f7be8101440 2023-11-17T00:05:48.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.805+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be810bb70 con 0x7f7be8101440 2023-11-17T00:05:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.809+0000 7f7be4ff9700 1 -- 172.21.15.23:0/419253579 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7bd8010580 con 0x7f7be8101440 2023-11-17T00:05:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.810+0000 7f7c01106b80 1 Processor -- start 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.810+0000 7f7c01106b80 1 -- start start 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7be4ff9700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 0x557d3b91f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7be4ff9700 1 -- 172.21.15.23:0/419253579 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f7bd80cc3a0 con 0x7f7be8101440 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9205d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b920b10 0x557d3b920f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b825000 con 0x557d3b77b6b0 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b877840 con 0x557d3b91fdb0 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.811+0000 7f7c01106b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b870cd0 con 0x557d3b920b10 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 0x557d3b91f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9205d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 0x557d3b91f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41588/0 (socket says 172.21.15.23:41588) 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be77fe700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7bdeffd700 1 -- 172.21.15.23:0/3694153065 learned_addr learned my addr 172.21.15.23:0/3694153065 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 -- 172.21.15.23:0/3694153065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b920b10 msgr2=0x557d3b920f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b920b10 0x557d3b920f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 -- 172.21.15.23:0/3694153065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 msgr2=0x557d3b91f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 0x557d3b91f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 -- 172.21.15.23:0/3694153065 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d3b6f25d0 con 0x557d3b91fdb0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.812+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9205d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80c9be0 tx=0x7f7bd8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fb20fc307a81ce server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.813+0000 7f7be77fe700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 0x7f7bd40a3600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7be000a400 tx=0x7f7be000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.813+0000 7f7bdd7fa700 1 -- 172.21.15.23:0/3694153065 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80994a0 con 0x557d3b91fdb0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.813+0000 7f7bdd7fa700 1 -- 172.21.15.23:0/3694153065 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bd809d4d0 con 0x557d3b91fdb0 2023-11-17T00:05:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.813+0000 7f7bdd7fa700 1 -- 172.21.15.23:0/3694153065 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd809d120 con 0x557d3b91fdb0 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 -- 172.21.15.23:0/3694153065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 msgr2=0x557d3b9205d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9205d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80c9be0 tx=0x7f7bd8004750 comp rx=0 tx=0).stop 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 -- 172.21.15.23:0/3694153065 shutdown_connections 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b920b10 0x557d3b920f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b77b6b0 0x557d3b91f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3694153065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9205d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 -- 172.21.15.23:0/3694153065 >> 172.21.15.23:0/3694153065 conn(0x557d3b914f50 msgr2=0x557d3b91a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.814+0000 7f7c01106b80 1 -- 172.21.15.23:0/3694153065 shutdown_connections 2023-11-17T00:05:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.815+0000 7f7c01106b80 1 -- 172.21.15.23:0/3694153065 wait complete. 2023-11-17T00:05:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.815+0000 7f7c01106b80 1 -- start start 2023-11-17T00:05:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.816+0000 7f7c01106b80 1 Processor -- start 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b812880 con 0x557d3b920b10 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b812520 con 0x557d3b77b6b0 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b6f48c0 con 0x557d3b91fdb0 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7c01106b80 10 client.0 _ll_register_callbacks cb 0x557d3b830720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7be77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7be77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41596/0 (socket says 172.21.15.23:41596) 2023-11-17T00:05:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7be77fe700 1 -- 172.21.15.23:0/1348313232 learned_addr learned my addr 172.21.15.23:0/1348313232 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.817+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b77b6b0 msgr2=0x557d3b928cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1348313232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 msgr2=0x557d3b930230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1348313232 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be810bb70 con 0x557d3b91fdb0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be77fe700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8099430 tx=0x7f7bd80b9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee21ecda323c6a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7bce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80994a0 con 0x557d3b91fdb0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f7bd809d120 con 0x557d3b91fdb0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.818+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d3b912710 con 0x557d3b91fdb0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bd809d2e0 con 0x557d3b91fdb0 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7c01106b80 10 client.25699 fetch_fsmap learned FSMap version 31 2023-11-17T00:05:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557d3b837130 con 0x557d3b91fdb0 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d3b912710 con 0x557d3b91fdb0 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7bd80a1460 con 0x557d3b91fdb0 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.819+0000 7f7c01106b80 10 client.25699 fetch_fsmap finished waiting for FSMap version 31 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7c01106b80 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7c01106b80 20 client.25699 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b9346d0 0x557d3b936ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7c01106b80 4 client.25699 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:05:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d3b833040 con 0x557d3b9346d0 2023-11-17T00:05:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.820+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b9346d0 0x557d3b936ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.821+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b9346d0 0x557d3b936ae0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14ead1fc35553f99 server_cookie=9348019e6ba6f5ef in_seq=0 out_seq=0 2023-11-17T00:05:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.821+0000 7f7bce7fc700 10 client.25699 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.821+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1348313232 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557d3b833040 con 0x557d3b9346d0 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.821+0000 7f7bce7fc700 10 client.25699 handle_command_reply: tid=0 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.821+0000 7f7c01106b80 1 client.25699 shutdown 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 msgr2=0x557d3b9291f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8099430 tx=0x7f7bd80b9040 comp rx=0 tx=0).stop 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 shutdown_connections 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b9346d0 0x557d3b936ae0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7bce7fc700 0 client.25699 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b930230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1348313232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 >> 172.21.15.23:0/1348313232 conn(0x557d3b914f50 msgr2=0x557d3b9197e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 shutdown_connections 2023-11-17T00:05:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 1 -- 172.21.15.23:0/1348313232 wait complete. 2023-11-17T00:05:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.822+0000 7f7c01106b80 20 client.25699 trim_cache size 0 max 16384 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.839+0000 7f7c01106b80 1 Processor -- start 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 -- start start 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b9301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b929be0 con 0x557d3b920b10 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b928190 con 0x557d3b91fdb0 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7c01106b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b920420 con 0x557d3b77b6b0 2023-11-17T00:05:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37142/0 (socket says 172.21.15.23:37142) 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7bdeffd700 1 -- 172.21.15.23:0/3760473117 learned_addr learned my addr 172.21.15.23:0/3760473117 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.840+0000 7f7be77fe700 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b9301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 -- 172.21.15.23:0/3760473117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 msgr2=0x557d3b9291f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 -- 172.21.15.23:0/3760473117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 msgr2=0x557d3b9301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b9301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 -- 172.21.15.23:0/3760473117 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d3b836e80 con 0x557d3b77b6b0 2023-11-17T00:05:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7be81012a0 tx=0x7f7bd0005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eac7fc52b3f0ea4f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.841+0000 7f7bddffb700 1 -- 172.21.15.23:0/3760473117 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd001d040 con 0x557d3b77b6b0 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.842+0000 7f7bddffb700 1 -- 172.21.15.23:0/3760473117 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bd001e040 con 0x557d3b77b6b0 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.842+0000 7f7bddffb700 1 -- 172.21.15.23:0/3760473117 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0027620 con 0x557d3b77b6b0 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- 172.21.15.23:0/3760473117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 msgr2=0x557d3b928cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7be81012a0 tx=0x7f7bd0005fb0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- 172.21.15.23:0/3760473117 shutdown_connections 2023-11-17T00:05:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b9301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b928cb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 --2- 172.21.15.23:0/3760473117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b9291f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- 172.21.15.23:0/3760473117 >> 172.21.15.23:0/3760473117 conn(0x557d3b914f50 msgr2=0x557d3b9279d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- 172.21.15.23:0/3760473117 shutdown_connections 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- 172.21.15.23:0/3760473117 wait complete. 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 -- start start 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.843+0000 7f7c01106b80 1 Processor -- start 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b92bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b92f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b92f970 con 0x557d3b920b10 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b92fae0 con 0x557d3b91fdb0 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3b92fca0 con 0x557d3b77b6b0 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7c01106b80 10 client.0 _ll_register_callbacks cb 0x7f7bd8099270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b92bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7be6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7be6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:40670/0 (socket says 172.21.15.23:40670) 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1953630118 learned_addr learned my addr 172.21.15.23:0/1953630118 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b92bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37150/0 (socket says 172.21.15.23:37150) 2023-11-17T00:05:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.844+0000 7f7be77fe700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b92f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1953630118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 msgr2=0x557d3b92bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b92bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1953630118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 msgr2=0x557d3b92f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b92f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 -- 172.21.15.23:0/1953630118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be8108fc0 con 0x557d3b91fdb0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7be6ffd700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8020780 tx=0x7f7bd8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b0038142ac9dc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7bce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8098e00 con 0x557d3b91fdb0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f7bd80978f0 con 0x557d3b91fdb0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bd809d4d0 con 0x557d3b91fdb0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bd80994a0 con 0x557d3b91fdb0 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7c01106b80 10 client.25682 fetch_fsmap learned FSMap version 31 2023-11-17T00:05:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.845+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557d3b687c00 con 0x557d3b91fdb0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bd80bf020 con 0x557d3b91fdb0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7bd8098990 con 0x557d3b91fdb0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 10 client.25682 fetch_fsmap finished waiting for FSMap version 31 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 20 client.25682 populate_metadata read hostname 'smithi023' 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b988ea0 0x557d3b98b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 4 client.25682 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.846+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557d3b83f600 con 0x557d3b988ea0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.847+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b988ea0 0x557d3b98b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.848+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b988ea0 0x557d3b98b2b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7698669b4490fd58 server_cookie=13399360850f50c4 in_seq=0 out_seq=0 2023-11-17T00:05:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:48.848+0000 7f7bce7fc700 10 client.25682 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.066+0000 7f7bce7fc700 1 -- 172.21.15.23:0/1953630118 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557d3b83f600 con 0x557d3b988ea0 2023-11-17T00:05:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.067+0000 7f7bce7fc700 10 client.25682 handle_command_reply: tid=0 2023-11-17T00:05:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.067+0000 7f7c01106b80 1 client.25682 shutdown 2023-11-17T00:05:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.068+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 msgr2=0x557d3b92c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.068+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8020780 tx=0x7f7bd8000bc0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 shutdown_connections 2023-11-17T00:05:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x557d3b988ea0 0x557d3b98b2b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d3b920b10 0x557d3b92f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557d3b77b6b0 0x557d3b92bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 --2- 172.21.15.23:0/1953630118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d3b91fdb0 0x557d3b92c410 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 >> 172.21.15.23:0/1953630118 conn(0x557d3b914f50 msgr2=0x557d3b928c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7bce7fc700 0 client.25682 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:05:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.069+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 shutdown_connections 2023-11-17T00:05:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.070+0000 7f7c01106b80 1 -- 172.21.15.23:0/1953630118 wait complete. 2023-11-17T00:05:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.072+0000 7f7c01106b80 20 client.25682 trim_cache size 0 max 16384 2023-11-17T00:05:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:05:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:05:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:05:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:05:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.074+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 msgr2=0x7f7bd40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.074+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 0x7f7bd40a3600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7be000a400 tx=0x7f7be000d040 comp rx=0 tx=0).stop 2023-11-17T00:05:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.074+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 msgr2=0x7f7be81945a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:05:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.074+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800b530 tx=0x7f7bd800b840 comp rx=0 tx=0).stop 2023-11-17T00:05:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 shutdown_connections 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7be8101dd0 0x7f7be81acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be8100250 0x7f7be8194060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7bd40a1150 0x7f7bd40a3600 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 --2- 172.21.15.23:0/419253579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7be8101440 0x7f7be81945a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 >> 172.21.15.23:0/419253579 conn(0x7f7be80fb850 msgr2=0x7f7be8106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 shutdown_connections 2023-11-17T00:05:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:05:49.075+0000 7f7bed7ff700 1 -- 172.21.15.23:0/419253579 wait complete. 2023-11-17T00:05:50.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:49 smithi173 ceph-mon[131534]: pgmap v511: 129 pgs: 129 active+clean; 242 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 556 op/s 2023-11-17T00:05:50.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:49 smithi156 ceph-mon[131105]: pgmap v511: 129 pgs: 129 active+clean; 242 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 556 op/s 2023-11-17T00:05:50.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:49 smithi023 ceph-mon[97824]: pgmap v511: 129 pgs: 129 active+clean; 242 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 556 op/s 2023-11-17T00:05:51.043 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:50 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service: Service RestartSec=10s expired, scheduling restart. 2023-11-17T00:05:51.043 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:50 smithi173 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service: Scheduled restart job, restart counter is at 1. 2023-11-17T00:05:51.043 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:50 smithi173 systemd[1]: Stopped Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:05:51.043 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:50 smithi173 systemd[1]: Starting Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:05:51.044 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:51 smithi173 podman[160640]: 2023-11-17T00:05:51.303 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:51 smithi173 bash[160640]: 1371223afd3d333dd9358ce8ab11d9c606032949ba819e8d0c0737ca5c5a6d80 2023-11-17T00:05:51.303 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:51 smithi173 ceph-mds[160681]: -- 172.21.15.173:0/2012429029 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b1e8afd520 con 0x55b1e8ad3000 2023-11-17T00:05:51.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: starting mds.l at 2023-11-17T00:05:51.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:05:51 smithi173 systemd[1]: Started Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:05:52.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:52 smithi023 ceph-mon[97824]: pgmap v512: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 537 op/s 2023-11-17T00:05:52.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:52 smithi023 ceph-mon[97824]: daemon mds.l restarted 2023-11-17T00:05:52.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:52 smithi023 ceph-mon[97824]: osdmap e93: 12 total, 12 up, 12 in 2023-11-17T00:05:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:52 smithi173 ceph-mon[131534]: pgmap v512: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 537 op/s 2023-11-17T00:05:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:52 smithi173 ceph-mon[131534]: daemon mds.l restarted 2023-11-17T00:05:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:52 smithi173 ceph-mon[131534]: osdmap e93: 12 total, 12 up, 12 in 2023-11-17T00:05:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:52 smithi156 ceph-mon[131105]: pgmap v512: 129 pgs: 129 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 537 op/s 2023-11-17T00:05:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:52 smithi156 ceph-mon[131105]: daemon mds.l restarted 2023-11-17T00:05:52.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:52 smithi156 ceph-mon[131105]: osdmap e93: 12 total, 12 up, 12 in 2023-11-17T00:05:53.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:53 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6832/2831566569,v1:172.21.15.173:6833/2831566569] up:boot 2023-11-17T00:05:53.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:53 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-17T00:05:53.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:53 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:05:53.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:53 smithi023 ceph-mon[97824]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:53.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:53 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6832/2831566569,v1:172.21.15.173:6833/2831566569] up:boot 2023-11-17T00:05:53.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:53 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-17T00:05:53.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:53 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:05:53.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:53 smithi156 ceph-mon[131105]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:53.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:53 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6832/2831566569,v1:172.21.15.173:6833/2831566569] up:boot 2023-11-17T00:05:53.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:53 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-11-17T00:05:53.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:53 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:05:53.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:53 smithi173 ceph-mon[131534]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-11-17T00:05:54.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:54 smithi173 ceph-mon[131534]: pgmap v514: 129 pgs: 129 active+clean; 254 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:05:54.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:54 smithi156 ceph-mon[131105]: pgmap v514: 129 pgs: 129 active+clean; 254 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:05:54.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:54 smithi023 ceph-mon[97824]: pgmap v514: 129 pgs: 129 active+clean; 254 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:05:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:56 smithi156 ceph-mon[131105]: pgmap v515: 129 pgs: 129 active+clean; 256 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 547 op/s 2023-11-17T00:05:56.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:56 smithi173 ceph-mon[131534]: pgmap v515: 129 pgs: 129 active+clean; 256 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 547 op/s 2023-11-17T00:05:56.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:56 smithi023 ceph-mon[97824]: pgmap v515: 129 pgs: 129 active+clean; 256 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 547 op/s 2023-11-17T00:05:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:58 smithi173 ceph-mon[131534]: pgmap v516: 129 pgs: 129 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 568 op/s 2023-11-17T00:05:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:58 smithi156 ceph-mon[131105]: pgmap v516: 129 pgs: 129 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 568 op/s 2023-11-17T00:05:58.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:58 smithi023 ceph-mon[97824]: pgmap v516: 129 pgs: 129 active+clean; 259 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 568 op/s 2023-11-17T00:06:00.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:05:59 smithi023 ceph-mon[97824]: pgmap v517: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 586 op/s 2023-11-17T00:06:08.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:05:59 smithi156 ceph-mon[131105]: pgmap v517: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 586 op/s 2023-11-17T00:06:08.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:02 smithi156 ceph-mon[131105]: pgmap v518: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 558 op/s 2023-11-17T00:06:08.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:04 smithi156 ceph-mon[131105]: pgmap v519: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 585 op/s 2023-11-17T00:06:08.055 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:06 smithi156 ceph-mon[131105]: pgmap v520: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:06:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:05:59 smithi173 ceph-mon[131534]: pgmap v517: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 586 op/s 2023-11-17T00:06:08.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:02 smithi173 ceph-mon[131534]: pgmap v518: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 558 op/s 2023-11-17T00:06:08.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:04 smithi173 ceph-mon[131534]: pgmap v519: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 585 op/s 2023-11-17T00:06:08.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:06 smithi173 ceph-mon[131534]: pgmap v520: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 -- 172.21.15.23:0/2906557670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 msgr2=0x7f7798107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 --2- 172.21.15.23:0/2906557670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f7798107120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7788009990 tx=0x7f7788009ca0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 -- 172.21.15.23:0/2906557670 shutdown_connections 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 --2- 172.21.15.23:0/2906557670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7798108400 0x7f779810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 --2- 172.21.15.23:0/2906557670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f7798107120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 --2- 172.21.15.23:0/2906557670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f7798107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.306+0000 7f779dfff700 1 -- 172.21.15.23:0/2906557670 >> 172.21.15.23:0/2906557670 conn(0x7f77980758d0 msgr2=0x7f7798077d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.307+0000 7f779dfff700 1 -- 172.21.15.23:0/2906557670 shutdown_connections 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.307+0000 7f779dfff700 1 -- 172.21.15.23:0/2906557670 wait complete. 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.308+0000 7f779dfff700 1 Processor -- start 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.309+0000 7f779dfff700 1 -- start start 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.309+0000 7f779dfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f779dfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f779819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f779dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7798108400 0x7f77981a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f779dfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981a4120 con 0x7f7798107a70 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f779dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981a4260 con 0x7f7798108400 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f779dfff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981a43f0 con 0x7f7798106d10 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37684/0 (socket says 172.21.15.23:37684) 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 -- 172.21.15.23:0/2593267797 learned_addr learned my addr 172.21.15.23:0/2593267797 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f7796ffd700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f779819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 -- 172.21.15.23:0/2593267797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7798108400 msgr2=0x7f77981a3b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7798108400 0x7f77981a3b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.310+0000 7f77977fe700 1 -- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 msgr2=0x7f779819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f77977fe700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f779819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f77977fe700 1 -- 172.21.15.23:0/2593267797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880095f0 con 0x7f7798106d10 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f7796ffd700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f779819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f77977fe700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7788003ad0 tx=0x7f7788003bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f7794ff9700 1 -- 172.21.15.23:0/2593267797 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778801d070 con 0x7f7798106d10 2023-11-17T00:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f7794ff9700 1 -- 172.21.15.23:0/2593267797 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f778800f460 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f7794ff9700 1 -- 172.21.15.23:0/2593267797 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788017650 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.311+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77981121d0 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.312+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7798112040 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.316+0000 7f7794ff9700 1 -- 172.21.15.23:0/2593267797 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f778800f5d0 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.317+0000 7f7794ff9700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 0x7f77800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.318+0000 7f7794ff9700 1 -- 172.21.15.23:0/2593267797 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f77880cde40 con 0x7f7798106d10 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.318+0000 7f7796ffd700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 0x7f77800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.318+0000 7f77b18fcb80 1 Processor -- start 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.318+0000 7f77b18fcb80 1 -- start start 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e5fad6b0 0x5558e6151870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e6152b10 0x5558e6152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6057000 con 0x5558e6152b10 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e60a9840 con 0x5558e5fad6b0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e60a2cd0 con 0x5558e6151db0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 0x7f77800a3680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f778c005fd0 tx=0x7f778c005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7797fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e6152b10 0x5558e6152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37694/0 (socket says 172.21.15.23:37694) 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 -- 172.21.15.23:0/3791819361 learned_addr learned my addr 172.21.15.23:0/3791819361 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f77977fe700 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e5fad6b0 0x5558e6151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 -- 172.21.15.23:0/3791819361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e5fad6b0 msgr2=0x5558e6151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e5fad6b0 0x5558e6151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 -- 172.21.15.23:0/3791819361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e6152b10 msgr2=0x5558e6152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e6152b10 0x5558e6152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.319+0000 7f7796ffd700 1 -- 172.21.15.23:0/3791819361 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e5f245d0 con 0x5558e6151db0 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.320+0000 7f7796ffd700 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f778c00edd0 tx=0x7f778c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be4f7f2f3112e2a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.320+0000 7f777dffb700 1 -- 172.21.15.23:0/3791819361 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c022070 con 0x5558e6151db0 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.320+0000 7f777dffb700 1 -- 172.21.15.23:0/3791819361 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f778c00ee00 con 0x5558e6151db0 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.320+0000 7f777dffb700 1 -- 172.21.15.23:0/3791819361 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c005440 con 0x5558e6151db0 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3791819361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 msgr2=0x5558e61525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f778c00edd0 tx=0x7f778c009f90 comp rx=0 tx=0).stop 2023-11-17T00:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3791819361 shutdown_connections 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e6152b10 0x5558e6152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e5fad6b0 0x5558e6151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3791819361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e61525d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3791819361 >> 172.21.15.23:0/3791819361 conn(0x5558e6146f50 msgr2=0x5558e614c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3791819361 shutdown_connections 2023-11-17T00:06:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.321+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3791819361 wait complete. 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.322+0000 7f77b18fcb80 1 -- start start 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.322+0000 7f77b18fcb80 1 Processor -- start 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e615b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 0x5558e61621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6044880 con 0x5558e5fad6b0 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6044520 con 0x5558e6152b10 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e5f268c0 con 0x5558e6151db0 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77b18fcb80 10 client.0 _ll_register_callbacks cb 0x5558e6062720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e615b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e615b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:37704/0 (socket says 172.21.15.23:37704) 2023-11-17T00:06:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7796ffd700 1 -- 172.21.15.23:0/3617506545 learned_addr learned my addr 172.21.15.23:0/3617506545 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f77977fe700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 0x5558e61621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 -- 172.21.15.23:0/3617506545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 msgr2=0x5558e615b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e615b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 -- 172.21.15.23:0/3617506545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 msgr2=0x5558e615ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.323+0000 7f7797fff700 1 -- 172.21.15.23:0/3617506545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c00eab0 con 0x5558e6152b10 2023-11-17T00:06:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f7797fff700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 0x5558e61621c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7790009ee0 tx=0x7f7790004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b57e5a881edc8aab server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f7776ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790019070 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558e6144710 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f7790004bb0 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7790004930 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f77b18fcb80 10 client.25694 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.324+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5558e6069130 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558e6144710 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f779001c020 con 0x5558e6152b10 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 10 client.25694 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 10 client.25694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 20 client.25694 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e6166ad0 0x5558e6168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 4 client.25694 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.325+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558e6065040 con 0x5558e6166ad0 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.326+0000 7f77977fe700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e6166ad0 0x5558e6168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.329+0000 7f77977fe700 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e6166ad0 0x5558e6168ee0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f61c8bd43fd1f783 server_cookie=c3b0de0212be8bcc in_seq=0 out_seq=0 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.330+0000 7f7776ffd700 10 client.25694 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.332+0000 7f7776ffd700 1 -- 172.21.15.23:0/3617506545 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5558e6065040 con 0x5558e6166ad0 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.332+0000 7f7776ffd700 10 client.25694 handle_command_reply: tid=0 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.332+0000 7f77b18fcb80 1 client.25694 shutdown 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.332+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 msgr2=0x5558e61621c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.332+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 0x5558e61621c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7790009ee0 tx=0x7f7790004320 comp rx=0 tx=0).stop 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 shutdown_connections 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6152b10 0x5558e61621c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e6166ad0 0x5558e6168ee0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3617506545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6151db0 0x5558e615b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 >> 172.21.15.23:0/3617506545 conn(0x5558e6146f50 msgr2=0x5558e614b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f7776ffd700 0 client.25694 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 shutdown_connections 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.333+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3617506545 wait complete. 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.334+0000 7f77b18fcb80 20 client.25694 trim_cache size 0 max 16384 2023-11-17T00:06:08.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.360+0000 7f77b18fcb80 1 Processor -- start 2023-11-17T00:06:08.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 -- start start 2023-11-17T00:06:08.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e61559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e6155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e61521c0 con 0x5558e5fad6b0 2023-11-17T00:06:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6152420 con 0x5558e6151db0 2023-11-17T00:06:08.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.361+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e614db90 con 0x5558e6152b10 2023-11-17T00:06:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.362+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.362+0000 7f7796ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:38434/0 (socket says 172.21.15.23:38434) 2023-11-17T00:06:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.362+0000 7f7796ffd700 1 -- 172.21.15.23:0/976987330 learned_addr learned my addr 172.21.15.23:0/976987330 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.362+0000 7f7797fff700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e6155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.363+0000 7f7796ffd700 1 -- 172.21.15.23:0/976987330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 msgr2=0x5558e6155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.363+0000 7f7796ffd700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e6155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.363+0000 7f7796ffd700 1 -- 172.21.15.23:0/976987330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 msgr2=0x5558e61559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.363+0000 7f77977fe700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e61559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.364+0000 7f7796ffd700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e61559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.364+0000 7f7796ffd700 1 -- 172.21.15.23:0/976987330 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e6068e80 con 0x5558e6151db0 2023-11-17T00:06:08.360 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.365+0000 7f77977fe700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e61559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-17T00:06:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.366+0000 7f7796ffd700 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f778c005f50 tx=0x7f778c011be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a321cc1b757d97 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.367+0000 7f777e7fc700 1 -- 172.21.15.23:0/976987330 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c022070 con 0x5558e6151db0 2023-11-17T00:06:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.367+0000 7f777e7fc700 1 -- 172.21.15.23:0/976987330 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f778c023040 con 0x5558e6151db0 2023-11-17T00:06:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.367+0000 7f777e7fc700 1 -- 172.21.15.23:0/976987330 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c004970 con 0x5558e6151db0 2023-11-17T00:06:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.368+0000 7f77b18fcb80 1 -- 172.21.15.23:0/976987330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 msgr2=0x5558e6162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.368+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f778c005f50 tx=0x7f778c011be0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.369+0000 7f77b18fcb80 1 -- 172.21.15.23:0/976987330 shutdown_connections 2023-11-17T00:06:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.369+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e6155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.369+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e61559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.369+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/976987330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e6151db0 0x5558e6162180 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f778c005f50 tx=0x7f778c011be0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.369+0000 7f77b18fcb80 1 -- 172.21.15.23:0/976987330 >> 172.21.15.23:0/976987330 conn(0x5558e6146f50 msgr2=0x5558e6158e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.370+0000 7f77b18fcb80 1 -- 172.21.15.23:0/976987330 shutdown_connections 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.370+0000 7f77b18fcb80 1 -- 172.21.15.23:0/976987330 wait complete. 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.370+0000 7f77b18fcb80 1 -- start start 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.371+0000 7f77b18fcb80 1 Processor -- start 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.374+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.374+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e615e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.374+0000 7f77b18fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e61612e0 0x5558e6161750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.375+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6161c90 con 0x5558e5fad6b0 2023-11-17T00:06:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.375+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6161e00 con 0x5558e61612e0 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.375+0000 7f77b18fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e6161fc0 con 0x5558e6152b10 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.375+0000 7f77b18fcb80 10 client.0 _ll_register_callbacks cb 0x7f7790003e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.375+0000 7f77977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.376+0000 7f77977fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35038/0 (socket says 172.21.15.23:35038) 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.376+0000 7f77977fe700 1 -- 172.21.15.23:0/3766481471 learned_addr learned my addr 172.21.15.23:0/3766481471 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.376+0000 7f7797fff700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e61612e0 0x5558e6161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.376+0000 7f77977fe700 1 -- 172.21.15.23:0/3766481471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 msgr2=0x5558e615e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.376+0000 7f77977fe700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e615e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.377+0000 7f77977fe700 1 -- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e61612e0 msgr2=0x5558e6161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.377+0000 7f77977fe700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e61612e0 0x5558e6161750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.377+0000 7f77977fe700 1 -- 172.21.15.23:0/3766481471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558e5f245d0 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.377+0000 7f77977fe700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f778809aec0 tx=0x7f7788009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6452e4976c98ed server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.377+0000 7f7776ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.378+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778800fab0 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.378+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f77880052d0 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.378+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f778c00ee00 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.379+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7788005490 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.379+0000 7f77b18fcb80 10 client.15651 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.379+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5558e5eb9c00 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.381+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f778c00ee00 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f77880a2e60 con 0x5558e5fad6b0 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 10 client.15651 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 10 client.15651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:08.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 20 client.15651 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e61bb890 0x5558e61bdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 4 client.15651 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.383+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5558e6071600 con 0x5558e61bb890 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.385+0000 7f7796ffd700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e61bb890 0x5558e61bdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.386+0000 7f7796ffd700 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e61bb890 0x5558e61bdca0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424dafedeb7cadc2 server_cookie=e0f75d8d1c63eaa7 in_seq=0 out_seq=0 2023-11-17T00:06:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:02.386+0000 7f7776ffd700 10 client.15651 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:08.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:02 smithi023 ceph-mon[97824]: pgmap v518: 129 pgs: 129 active+clean; 265 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 558 op/s 2023-11-17T00:06:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.195+0000 7f7776ffd700 1 -- 172.21.15.23:0/3766481471 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5558e6071600 con 0x5558e61bb890 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.195+0000 7f7776ffd700 10 client.15651 handle_command_reply: tid=0 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.195+0000 7f77b18fcb80 1 client.15651 shutdown 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 msgr2=0x5558e615dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f778809aec0 tx=0x7f7788009ca0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 shutdown_connections 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5558e61612e0 0x5558e6161750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5558e61bb890 0x5558e61bdca0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5558e5fad6b0 0x5558e615dd80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 --2- 172.21.15.23:0/3766481471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558e6152b10 0x5558e615e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 >> 172.21.15.23:0/3766481471 conn(0x5558e6146f50 msgr2=0x5558e6158e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.196+0000 7f7776ffd700 0 client.15651 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.197+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 shutdown_connections 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.197+0000 7f77b18fcb80 1 -- 172.21.15.23:0/3766481471 wait complete. 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.197+0000 7f77b18fcb80 20 client.15651 trim_cache size 0 max 16384 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 msgr2=0x7f77800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 0x7f77800a3680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f778c005fd0 tx=0x7f778c005ee0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 msgr2=0x7f779819c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7788003ad0 tx=0x7f7788003bb0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 shutdown_connections 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7798108400 0x7f77981a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f77800a11d0 0x7f77800a3680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7798106d10 0x7f779819c5a0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7788003ad0 tx=0x7f7788003bb0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 --2- 172.21.15.23:0/2593267797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7798107a70 0x7f779819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 >> 172.21.15.23:0/2593267797 conn(0x7f77980758d0 msgr2=0x7f779810af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 shutdown_connections 2023-11-17T00:06:08.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:03.198+0000 7f779dfff700 1 -- 172.21.15.23:0/2593267797 wait complete. 2023-11-17T00:06:08.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:04 smithi023 ceph-mon[97824]: pgmap v519: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 585 op/s 2023-11-17T00:06:08.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:06 smithi023 ceph-mon[97824]: pgmap v520: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 562 op/s 2023-11-17T00:06:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:08 smithi023 ceph-mon[97824]: pgmap v521: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 510 op/s 2023-11-17T00:06:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:08 smithi173 ceph-mon[131534]: pgmap v521: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 510 op/s 2023-11-17T00:06:08.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:08 smithi156 ceph-mon[131105]: pgmap v521: 129 pgs: 129 active+clean; 273 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 510 op/s 2023-11-17T00:06:10.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:09 smithi023 ceph-mon[97824]: pgmap v522: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 542 op/s 2023-11-17T00:06:10.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:09 smithi173 ceph-mon[131534]: pgmap v522: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 542 op/s 2023-11-17T00:06:10.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:09 smithi156 ceph-mon[131105]: pgmap v522: 129 pgs: 129 active+clean; 277 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 542 op/s 2023-11-17T00:06:12.519 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:12 smithi023 ceph-mon[97824]: pgmap v523: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 508 op/s 2023-11-17T00:06:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:12 smithi156 ceph-mon[131105]: pgmap v523: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 508 op/s 2023-11-17T00:06:12.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:12 smithi173 ceph-mon[131534]: pgmap v523: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 508 op/s 2023-11-17T00:06:14.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:14 smithi023 ceph-mon[97824]: pgmap v524: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 571 op/s 2023-11-17T00:06:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:14 smithi173 ceph-mon[131534]: pgmap v524: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 571 op/s 2023-11-17T00:06:14.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:14 smithi156 ceph-mon[131105]: pgmap v524: 129 pgs: 129 active+clean; 283 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 571 op/s 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 -- 172.21.15.23:0/200460423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409d520 msgr2=0x7fc8140a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 --2- 172.21.15.23:0/200460423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409d520 0x7fc8140a3de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc80801a9a0 tx=0x7fc80801acb0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 -- 172.21.15.23:0/200460423 shutdown_connections 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 --2- 172.21.15.23:0/200460423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409d520 0x7fc8140a3de0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 --2- 172.21.15.23:0/200460423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc81409b9a0 0x7fc81409bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 --2- 172.21.15.23:0/200460423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409cb90 0x7fc81409cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 -- 172.21.15.23:0/200460423 >> 172.21.15.23:0/200460423 conn(0x7fc81400a9d0 msgr2=0x7fc814099790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.500+0000 7fc821b9d700 1 -- 172.21.15.23:0/200460423 shutdown_connections 2023-11-17T00:06:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.501+0000 7fc821b9d700 1 -- 172.21.15.23:0/200460423 wait complete. 2023-11-17T00:06:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.503+0000 7fc821b9d700 1 Processor -- start 2023-11-17T00:06:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.504+0000 7fc821b9d700 1 -- start start 2023-11-17T00:06:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.504+0000 7fc821b9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 0x7fc814131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.504+0000 7fc821b9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc821b9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc81409d520 0x7fc814138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 0x7fc814131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 0x7fc814131810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39018/0 (socket says 172.21.15.23:39018) 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc821b9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc814139390 con 0x7fc81409d520 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc821b9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8141394d0 con 0x7fc81409cb90 2023-11-17T00:06:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc821b9d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc814139660 con 0x7fc81409b9a0 2023-11-17T00:06:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc81bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc81bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34194/0 (socket says 172.21.15.23:34194) 2023-11-17T00:06:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.505+0000 7fc81bfff700 1 -- 172.21.15.23:0/2264840501 learned_addr learned my addr 172.21.15.23:0/2264840501 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 -- 172.21.15.23:0/2264840501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 msgr2=0x7fc814131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 0x7fc814131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 -- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc81409d520 msgr2=0x7fc814138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc81409d520 0x7fc814138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 -- 172.21.15.23:0/2264840501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80801a600 con 0x7fc81409cb90 2023-11-17T00:06:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.506+0000 7fc81bfff700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc81000b550 tx=0x7fc81000b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.507+0000 7fc819ffb700 1 -- 172.21.15.23:0/2264840501 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810011830 con 0x7fc81409cb90 2023-11-17T00:06:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.507+0000 7fc819ffb700 1 -- 172.21.15.23:0/2264840501 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc810011e50 con 0x7fc81409cb90 2023-11-17T00:06:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.507+0000 7fc819ffb700 1 -- 172.21.15.23:0/2264840501 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81000f2e0 con 0x7fc81409cb90 2023-11-17T00:06:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.507+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8140a7490 con 0x7fc81409cb90 2023-11-17T00:06:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.508+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8140a7300 con 0x7fc81409cb90 2023-11-17T00:06:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.516+0000 7fc819ffb700 1 -- 172.21.15.23:0/2264840501 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc810010380 con 0x7fc81409cb90 2023-11-17T00:06:16.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:16 smithi023 ceph-mon[97824]: pgmap v525: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 600 op/s 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.519+0000 7fc819ffb700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 0x7fc80c0a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.520+0000 7fc819ffb700 1 -- 172.21.15.23:0/2264840501 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc8100cc5c0 con 0x7fc81409cb90 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.520+0000 7fc83679cb80 1 Processor -- start 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.520+0000 7fc820b9b700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 0x7fc80c0a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.521+0000 7fc83679cb80 1 -- start start 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.521+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 0x55c50c7c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.522+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.522+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7c8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.522+0000 7fc83679cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c6cc000 con 0x55c50c6226b0 2023-11-17T00:06:16.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.522+0000 7fc83679cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c71e840 con 0x55c50c7c6fb0 2023-11-17T00:06:16.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.522+0000 7fc83679cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c717cd0 con 0x55c50c7c7d60 2023-11-17T00:06:16.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.523+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 0x55c50c7c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.523+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 0x55c50c7c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51514/0 (socket says 172.21.15.23:51514) 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.523+0000 7fc820b9b700 1 -- 172.21.15.23:0/79385415 learned_addr learned my addr 172.21.15.23:0/79385415 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.523+0000 7fc81bfff700 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.523+0000 7fc82139c700 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 -- 172.21.15.23:0/79385415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 msgr2=0x55c50c7c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 -- 172.21.15.23:0/79385415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 msgr2=0x55c50c7c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 0x55c50c7c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 -- 172.21.15.23:0/79385415 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c50c5995d0 con 0x55c50c7c7d60 2023-11-17T00:06:16.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.524+0000 7fc82139c700 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7c8200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc80801a8f0 tx=0x7fc80801c550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4110eb7e7798125 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.526+0000 7fc806ffd700 1 -- 172.21.15.23:0/79385415 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80801caf0 con 0x55c50c7c7d60 2023-11-17T00:06:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.526+0000 7fc806ffd700 1 -- 172.21.15.23:0/79385415 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8080056e0 con 0x55c50c7c7d60 2023-11-17T00:06:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.527+0000 7fc806ffd700 1 -- 172.21.15.23:0/79385415 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808030400 con 0x55c50c7c7d60 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.528+0000 7fc83679cb80 1 -- 172.21.15.23:0/79385415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 msgr2=0x55c50c7c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.528+0000 7fc83679cb80 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7c8200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc80801a8f0 tx=0x7fc80801c550 comp rx=0 tx=0).stop 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc820b9b700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 0x7fc80c0a3800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc81c066780 tx=0x7fc81c06ad10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc83679cb80 1 -- 172.21.15.23:0/79385415 shutdown_connections 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc83679cb80 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7c8200 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc83679cb80 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c6226b0 0x55c50c7c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc83679cb80 1 --2- 172.21.15.23:0/79385415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.529+0000 7fc83679cb80 1 -- 172.21.15.23:0/79385415 >> 172.21.15.23:0/79385415 conn(0x55c50c7bc1a0 msgr2=0x55c50c7c12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.530+0000 7fc83679cb80 1 -- 172.21.15.23:0/79385415 shutdown_connections 2023-11-17T00:06:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.530+0000 7fc83679cb80 1 -- 172.21.15.23:0/79385415 wait complete. 2023-11-17T00:06:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.531+0000 7fc83679cb80 1 -- start start 2023-11-17T00:06:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.533+0000 7fc83679cb80 1 Processor -- start 2023-11-17T00:06:16.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c6b9880 con 0x55c50c7c7d60 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c6b9520 con 0x55c50c6226b0 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c59b8c0 con 0x55c50c7c6fb0 2023-11-17T00:06:16.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.535+0000 7fc83679cb80 10 client.0 _ll_register_callbacks cb 0x55c50c6d7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc82139c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34226/0 (socket says 172.21.15.23:34226) 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc820b9b700 1 -- 172.21.15.23:0/1218591274 learned_addr learned my addr 172.21.15.23:0/1218591274 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc820b9b700 1 -- 172.21.15.23:0/1218591274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 msgr2=0x55c50c7d03a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.536+0000 7fc820b9b700 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d03a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.537+0000 7fc820b9b700 1 -- 172.21.15.23:0/1218591274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 msgr2=0x55c50c7d73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.537+0000 7fc820b9b700 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.537+0000 7fc820b9b700 1 -- 172.21.15.23:0/1218591274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8140a4710 con 0x55c50c6226b0 2023-11-17T00:06:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.537+0000 7fc820b9b700 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c04fde0 tx=0x7fc81c063db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b47fc60b8f49e829 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.537+0000 7fc7fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:06:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.538+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c07e2e0 con 0x55c50c6226b0 2023-11-17T00:06:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.538+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc81c07e930 con 0x55c50c6226b0 2023-11-17T00:06:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.538+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c50c7b9960 con 0x55c50c6226b0 2023-11-17T00:06:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.538+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc81c07eaf0 con 0x55c50c6226b0 2023-11-17T00:06:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.540+0000 7fc83679cb80 10 client.25712 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.540+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c50c6de130 con 0x55c50c6226b0 2023-11-17T00:06:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.540+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c50c7b9960 con 0x55c50c6226b0 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.540+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc81c087450 con 0x55c50c6226b0 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 10 client.25712 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 20 client.25712 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c7dbd20 0x55c50c7de130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 4 client.25712 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:06:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c50c6da040 con 0x55c50c7dbd20 2023-11-17T00:06:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.541+0000 7fc81bfff700 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c7dbd20 0x55c50c7de130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.543+0000 7fc81bfff700 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c7dbd20 0x55c50c7de130 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a20538fd857d9f server_cookie=624296cbfa6ba801 in_seq=0 out_seq=0 2023-11-17T00:06:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.543+0000 7fc7fbfff700 10 client.25712 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.548+0000 7fc7fbfff700 1 -- 172.21.15.23:0/1218591274 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c50c6da040 con 0x55c50c7dbd20 2023-11-17T00:06:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.548+0000 7fc7fbfff700 10 client.25712 handle_command_reply: tid=0 2023-11-17T00:06:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.548+0000 7fc83679cb80 1 client.25712 shutdown 2023-11-17T00:06:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.550+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 msgr2=0x55c50c7cfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.550+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c04fde0 tx=0x7fc81c063db0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 shutdown_connections 2023-11-17T00:06:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c7dbd20 0x55c50c7de130 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c6226b0 0x55c50c7cfe60 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c04fde0 tx=0x7fc81c063db0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1218591274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 >> 172.21.15.23:0/1218591274 conn(0x55c50c7bc1a0 msgr2=0x55c50c7c12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc7fbfff700 0 client.25712 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 shutdown_connections 2023-11-17T00:06:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.551+0000 7fc83679cb80 1 -- 172.21.15.23:0/1218591274 wait complete. 2023-11-17T00:06:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.552+0000 7fc83679cb80 20 client.25712 trim_cache size 0 max 16384 2023-11-17T00:06:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.595+0000 7fc83679cb80 1 Processor -- start 2023-11-17T00:06:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.596+0000 7fc83679cb80 1 -- start start 2023-11-17T00:06:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.596+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.596+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.596+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 0x55c50c7cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc81bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc81bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39048/0 (socket says 172.21.15.23:39048) 2023-11-17T00:06:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:34236/0 (socket says 172.21.15.23:34236) 2023-11-17T00:06:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc81bfff700 1 -- 172.21.15.23:0/1975082744 learned_addr learned my addr 172.21.15.23:0/1975082744 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7bbd70 con 0x55c50c7d73a0 2023-11-17T00:06:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7cde30 con 0x55c50c7c6fb0 2023-11-17T00:06:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.597+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7cb740 con 0x55c50c7c7d60 2023-11-17T00:06:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 0x55c50c7cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 -- 172.21.15.23:0/1975082744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 msgr2=0x55c50c7ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 -- 172.21.15.23:0/1975082744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 msgr2=0x55c50c7ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.598+0000 7fc82139c700 1 -- 172.21.15.23:0/1975082744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c50c6dde80 con 0x55c50c7d73a0 2023-11-17T00:06:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.600+0000 7fc82139c700 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 0x55c50c7cabd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55c50c7c19c0 tx=0x7fc80801af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c748ed7d6c9f3baf server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.602+0000 7fc8077fe700 1 -- 172.21.15.23:0/1975082744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808005320 con 0x55c50c7d73a0 2023-11-17T00:06:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.602+0000 7fc8077fe700 1 -- 172.21.15.23:0/1975082744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc80801ce50 con 0x55c50c7d73a0 2023-11-17T00:06:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.602+0000 7fc8077fe700 1 -- 172.21.15.23:0/1975082744 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80803b680 con 0x55c50c7d73a0 2023-11-17T00:06:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.603+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 msgr2=0x55c50c7cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.603+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 0x55c50c7cabd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55c50c7c19c0 tx=0x7fc80801af10 comp rx=0 tx=0).stop 2023-11-17T00:06:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 shutdown_connections 2023-11-17T00:06:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7d73a0 0x55c50c7cabd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7c6fb0 0x55c50c7ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 --2- 172.21.15.23:0/1975082744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c7d60 0x55c50c7ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 >> 172.21.15.23:0/1975082744 conn(0x55c50c7bc1a0 msgr2=0x55c50c7cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.604+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 shutdown_connections 2023-11-17T00:06:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.605+0000 7fc83679cb80 1 -- 172.21.15.23:0/1975082744 wait complete. 2023-11-17T00:06:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.605+0000 7fc83679cb80 1 -- start start 2023-11-17T00:06:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:16 smithi156 ceph-mon[131105]: pgmap v525: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 600 op/s 2023-11-17T00:06:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.606+0000 7fc83679cb80 1 Processor -- start 2023-11-17T00:06:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:16 smithi173 ceph-mon[131534]: pgmap v525: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 600 op/s 2023-11-17T00:06:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7d73a0 0x55c50c7d5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7d6610 con 0x55c50c7c7d60 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7d6780 con 0x55c50c7d73a0 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50c7d6940 con 0x55c50c7c6fb0 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.607+0000 7fc83679cb80 10 client.0 _ll_register_callbacks cb 0x55c50c6d7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.608+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.608+0000 7fc820b9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39056/0 (socket says 172.21.15.23:39056) 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.608+0000 7fc820b9b700 1 -- 172.21.15.23:0/3882415281 learned_addr learned my addr 172.21.15.23:0/3882415281 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.608+0000 7fc82139c700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7d73a0 0x55c50c7d5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.609+0000 7fc820b9b700 1 -- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7d73a0 msgr2=0x55c50c7d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.609+0000 7fc820b9b700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7d73a0 0x55c50c7d5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.609+0000 7fc820b9b700 1 -- 172.21.15.23:0/3882415281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 msgr2=0x55c50c7d58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.609+0000 7fc820b9b700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.609+0000 7fc820b9b700 1 -- 172.21.15.23:0/3882415281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c50c5995d0 con 0x55c50c7c6fb0 2023-11-17T00:06:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.610+0000 7fc820b9b700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0877e0 tx=0x7fc81c07e390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e66bf6765a575e99 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.610+0000 7fc7fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2023-11-17T00:06:16.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.611+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c04ef20 con 0x55c50c7c6fb0 2023-11-17T00:06:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.611+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fc81c06a040 con 0x55c50c7c6fb0 2023-11-17T00:06:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.611+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8080056e0 con 0x55c50c7c6fb0 2023-11-17T00:06:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.613+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc81c06a200 con 0x55c50c7c6fb0 2023-11-17T00:06:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.613+0000 7fc83679cb80 10 client.25735 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.613+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c50c52ec00 con 0x55c50c7c6fb0 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8080056e0 con 0x55c50c7c6fb0 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc81c09d070 con 0x55c50c7c6fb0 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc83679cb80 10 client.25735 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc83679cb80 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc83679cb80 20 client.25735 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.616+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c830a80 0x55c50c832e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.617+0000 7fc83679cb80 4 client.25735 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.617+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c50c6e6600 con 0x55c50c830a80 2023-11-17T00:06:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.617+0000 7fc81bfff700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c830a80 0x55c50c832e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.618+0000 7fc7fbfff700 10 client.25735 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.619+0000 7fc81bfff700 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c830a80 0x55c50c832e90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf4dd34ebfde5ed server_cookie=8fa10404e99c1751 in_seq=0 out_seq=0 2023-11-17T00:06:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.926+0000 7fc7fbfff700 1 -- 172.21.15.23:0/3882415281 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c50c6e6600 con 0x55c50c830a80 2023-11-17T00:06:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.926+0000 7fc7fbfff700 10 client.25735 handle_command_reply: tid=0 2023-11-17T00:06:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.926+0000 7fc83679cb80 1 client.25735 shutdown 2023-11-17T00:06:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 msgr2=0x55c50c7d5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0877e0 tx=0x7fc81c07e390 comp rx=0 tx=0).stop 2023-11-17T00:06:16.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 shutdown_connections 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c50c7d73a0 0x55c50c7d5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c50c7c6fb0 0x55c50c7d5390 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55c50c830a80 0x55c50c832e90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 --2- 172.21.15.23:0/3882415281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c50c7c7d60 0x55c50c7d58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 >> 172.21.15.23:0/3882415281 conn(0x55c50c7bc1a0 msgr2=0x55c50c7ce080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.927+0000 7fc7fbfff700 0 client.25735 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.928+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 shutdown_connections 2023-11-17T00:06:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.930+0000 7fc83679cb80 1 -- 172.21.15.23:0/3882415281 wait complete. 2023-11-17T00:06:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.930+0000 7fc83679cb80 20 client.25735 trim_cache size 0 max 16384 2023-11-17T00:06:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:06:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:06:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:06:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 msgr2=0x7fc80c0a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 0x7fc80c0a3800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc81c066780 tx=0x7fc81c06ad10 comp rx=0 tx=0).stop 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 msgr2=0x7fc814131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc81000b550 tx=0x7fc81000b860 comp rx=0 tx=0).stop 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 shutdown_connections 2023-11-17T00:06:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc81409d520 0x7fc814138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fc80c0a1350 0x7fc80c0a3800 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc81409b9a0 0x7fc814131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 --2- 172.21.15.23:0/2264840501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc81409cb90 0x7fc814131d50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.931+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 >> 172.21.15.23:0/2264840501 conn(0x7fc81400a9d0 msgr2=0x7fc8140a0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.932+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 shutdown_connections 2023-11-17T00:06:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:16.932+0000 7fc821b9d700 1 -- 172.21.15.23:0/2264840501 wait complete. 2023-11-17T00:06:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:18 smithi023 ceph-mon[97824]: pgmap v526: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2023-11-17T00:06:18.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:18 smithi173 ceph-mon[131534]: pgmap v526: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2023-11-17T00:06:18.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:18 smithi156 ceph-mon[131105]: pgmap v526: 129 pgs: 129 active+clean; 293 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2023-11-17T00:06:20.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:19 smithi023 ceph-mon[97824]: pgmap v527: 129 pgs: 129 active+clean; 292 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 609 op/s 2023-11-17T00:06:20.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:19 smithi173 ceph-mon[131534]: pgmap v527: 129 pgs: 129 active+clean; 292 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 609 op/s 2023-11-17T00:06:20.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:19 smithi156 ceph-mon[131105]: pgmap v527: 129 pgs: 129 active+clean; 292 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 609 op/s 2023-11-17T00:06:22.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:22 smithi023 ceph-mon[97824]: pgmap v528: 129 pgs: 129 active+clean; 298 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2023-11-17T00:06:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:22 smithi173 ceph-mon[131534]: pgmap v528: 129 pgs: 129 active+clean; 298 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2023-11-17T00:06:22.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:22 smithi156 ceph-mon[131105]: pgmap v528: 129 pgs: 129 active+clean; 298 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2023-11-17T00:06:24.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:24 smithi023 ceph-mon[97824]: pgmap v529: 129 pgs: 129 active+clean; 298 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 655 op/s 2023-11-17T00:06:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:24 smithi173 ceph-mon[131534]: pgmap v529: 129 pgs: 129 active+clean; 298 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 655 op/s 2023-11-17T00:06:24.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:24 smithi156 ceph-mon[131105]: pgmap v529: 129 pgs: 129 active+clean; 298 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 655 op/s 2023-11-17T00:06:26.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:26 smithi023 ceph-mon[97824]: pgmap v530: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 618 op/s 2023-11-17T00:06:26.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:26 smithi156 ceph-mon[131105]: pgmap v530: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 618 op/s 2023-11-17T00:06:26.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:26 smithi173 ceph-mon[131534]: pgmap v530: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 618 op/s 2023-11-17T00:06:28.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:28 smithi023 ceph-mon[97824]: pgmap v531: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 509 op/s 2023-11-17T00:06:28.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:28 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:06:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:28 smithi173 ceph-mon[131534]: pgmap v531: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 509 op/s 2023-11-17T00:06:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:28 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:06:28.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:28 smithi156 ceph-mon[131105]: pgmap v531: 129 pgs: 129 active+clean; 305 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 509 op/s 2023-11-17T00:06:28.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:28 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:06:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.113+0000 7fd6f8fff700 1 -- 172.21.15.23:0/515616216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f40734d0 msgr2=0x7fd6f4073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.113+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/515616216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f40734d0 0x7fd6f4073940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0098f0 tx=0x7fd6dc009c00 comp rx=0 tx=0).stop 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 -- 172.21.15.23:0/515616216 shutdown_connections 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/515616216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 0x7fd6f410a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/515616216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4072f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/515616216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f40734d0 0x7fd6f4073940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 -- 172.21.15.23:0/515616216 >> 172.21.15.23:0/515616216 conn(0x7fd6f40fb8e0 msgr2=0x7fd6f40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 -- 172.21.15.23:0/515616216 shutdown_connections 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.114+0000 7fd6f8fff700 1 -- 172.21.15.23:0/515616216 wait complete. 2023-11-17T00:06:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 Processor -- start 2023-11-17T00:06:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 -- start start 2023-11-17T00:06:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f40734d0 0x7fd6f4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 0x7fd6f419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f419fda0 con 0x7fd6f40734d0 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f419fee0 con 0x7fd6f4074ab0 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.115+0000 7fd6f8fff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f41a0070 con 0x7fd6f4108170 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f40734d0 0x7fd6f4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42532/0 (socket says 172.21.15.23:42532) 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 -- 172.21.15.23:0/1127943076 learned_addr learned my addr 172.21.15.23:0/1127943076 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 0x7fd6f419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 -- 172.21.15.23:0/1127943076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 msgr2=0x7fd6f419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 0x7fd6f419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 -- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f40734d0 msgr2=0x7fd6f4198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f40734d0 0x7fd6f4198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 -- 172.21.15.23:0/1127943076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009520 con 0x7fd6f4074ab0 2023-11-17T00:06:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.116+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009fd0 tx=0x7fd6dc00b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.117+0000 7fd6eb7fe700 1 -- 172.21.15.23:0/1127943076 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc01c070 con 0x7fd6f4074ab0 2023-11-17T00:06:30.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.117+0000 7fd6eb7fe700 1 -- 172.21.15.23:0/1127943076 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc00f7d0 con 0x7fd6f4074ab0 2023-11-17T00:06:30.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.117+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f410ddf0 con 0x7fd6f4074ab0 2023-11-17T00:06:30.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.117+0000 7fd6eb7fe700 1 -- 172.21.15.23:0/1127943076 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc004ab0 con 0x7fd6f4074ab0 2023-11-17T00:06:30.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.117+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f410dc60 con 0x7fd6f4074ab0 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.120+0000 7fd6eb7fe700 1 -- 172.21.15.23:0/1127943076 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd6dc020070 con 0x7fd6f4074ab0 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.121+0000 7fd6eb7fe700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 0x7fd6e00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.122+0000 7fd6eb7fe700 1 -- 172.21.15.23:0/1127943076 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fd6dc0cd890 con 0x7fd6f4074ab0 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.122+0000 7fd6f259c700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 0x7fd6e00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.122+0000 7fd70c8fcb80 1 Processor -- start 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.122+0000 7fd70c8fcb80 1 -- start start 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f1e646b0 0x55d3f2008870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2009f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1f0e000 con 0x55d3f1e646b0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1f60840 con 0x55d3f2008db0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1f59cd0 con 0x55d3f2009b10 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6f259c700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 0x7fd6e00a3680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4005fd0 tx=0x7fd6e4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f1e646b0 0x55d3f2008870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42534/0 (socket says 172.21.15.23:42534) 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 -- 172.21.15.23:0/4166876532 learned_addr learned my addr 172.21.15.23:0/4166876532 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2009f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 -- 172.21.15.23:0/4166876532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 msgr2=0x55d3f2009f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2009f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 -- 172.21.15.23:0/4166876532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f1e646b0 msgr2=0x55d3f2008870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f1e646b0 0x55d3f2008870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.123+0000 7fd6ebfff700 1 -- 172.21.15.23:0/4166876532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f1ddb5d0 con 0x55d3f2008db0 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.124+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc097510 tx=0x7fd6dc09a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a729ce053684d51a server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.124+0000 7fd6e8ff9700 1 -- 172.21.15.23:0/4166876532 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc09ead0 con 0x55d3f2008db0 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.124+0000 7fd6e8ff9700 1 -- 172.21.15.23:0/4166876532 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc0a2e60 con 0x55d3f2008db0 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.124+0000 7fd6e8ff9700 1 -- 172.21.15.23:0/4166876532 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc09af40 con 0x55d3f2008db0 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4166876532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 msgr2=0x55d3f20095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc097510 tx=0x7fd6dc09a660 comp rx=0 tx=0).stop 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4166876532 shutdown_connections 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2009f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f1e646b0 0x55d3f2008870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4166876532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f20095d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4166876532 >> 172.21.15.23:0/4166876532 conn(0x55d3f1ffdf50 msgr2=0x55d3f2003100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4166876532 shutdown_connections 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4166876532 wait complete. 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.125+0000 7fd70c8fcb80 1 -- start start 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 Processor -- start 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2008db0 0x55d3f2011e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2018eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1efb880 con 0x55d3f2008db0 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1efb520 con 0x55d3f1e646b0 2023-11-17T00:06:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1ddd8c0 con 0x55d3f2009b10 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd70c8fcb80 10 client.0 _ll_register_callbacks cb 0x55d3f1f19720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42538/0 (socket says 172.21.15.23:42538) 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd6f259c700 1 -- 172.21.15.23:0/3047488739 learned_addr learned my addr 172.21.15.23:0/3047488739 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.126+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2008db0 0x55d3f2011e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2018eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 -- 172.21.15.23:0/3047488739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 msgr2=0x55d3f2018eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2018eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 -- 172.21.15.23:0/3047488739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2008db0 msgr2=0x55d3f2011e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2008db0 0x55d3f2011e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 -- 172.21.15.23:0/3047488739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f410dc60 con 0x55d3f1e646b0 2023-11-17T00:06:30.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6f259c700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003930 tx=0x7fd6e40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5b826746307039 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4004a30 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fd6e4004b90 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.127+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3f1ffb710 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e4004d50 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 10 client.25730 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d3f1f20130 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d3f1ffb710 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd6e401fb70 con 0x55d3f1e646b0 2023-11-17T00:06:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 10 client.25730 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 10 client.25730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 20 client.25730 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f201d810 0x55d3f201fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 4 client.25730 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.128+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3f1f1c040 con 0x55d3f201d810 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.129+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f201d810 0x55d3f201fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.129+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f201d810 0x55d3f201fc20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25b5284699faa39 server_cookie=eccc7e5c17c95fa0 in_seq=0 out_seq=0 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.129+0000 7fd6d9ffb700 10 client.25730 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/3047488739 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d3f1f1c040 con 0x55d3f201d810 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd6d9ffb700 10 client.25730 handle_command_reply: tid=0 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 client.25730 shutdown 2023-11-17T00:06:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 msgr2=0x55d3f2011930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003930 tx=0x7fd6e40047c0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 shutdown_connections 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2009b10 0x55d3f2018eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f2011930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f201d810 0x55d3f201fc20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3047488739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2008db0 0x55d3f2011e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 >> 172.21.15.23:0/3047488739 conn(0x55d3f1ffdf50 msgr2=0x55d3f20027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd6d9ffb700 0 client.25730 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 shutdown_connections 2023-11-17T00:06:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3047488739 wait complete. 2023-11-17T00:06:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.130+0000 7fd70c8fcb80 20 client.25730 trim_cache size 0 max 16384 2023-11-17T00:06:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.146+0000 7fd70c8fcb80 1 Processor -- start 2023-11-17T00:06:30.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 -- start start 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 0x55d3f2024540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f20116e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2012ab0 con 0x55d3f2009b10 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2010de0 con 0x55d3f2008db0 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2009420 con 0x55d3f1e646b0 2023-11-17T00:06:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42540/0 (socket says 172.21.15.23:42540) 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 -- 172.21.15.23:0/3163217524 learned_addr learned my addr 172.21.15.23:0/3163217524 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f20116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6f259c700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 0x55d3f2024540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 -- 172.21.15.23:0/3163217524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 msgr2=0x55d3f2024540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 0x55d3f2024540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 -- 172.21.15.23:0/3163217524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 msgr2=0x55d3f20116e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f20116e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6ebfff700 1 -- 172.21.15.23:0/3163217524 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f1f1fe80 con 0x55d3f2008db0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.147+0000 7fd6f259c700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 0x55d3f2024540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.148+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc09aa30 tx=0x7fd6dc09a5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a29af3ee4e17df server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.148+0000 7fd6e97fa700 1 -- 172.21.15.23:0/3163217524 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc09eb50 con 0x55d3f2008db0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.148+0000 7fd6e97fa700 1 -- 172.21.15.23:0/3163217524 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc004c10 con 0x55d3f2008db0 2023-11-17T00:06:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.148+0000 7fd6e97fa700 1 -- 172.21.15.23:0/3163217524 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc09ad00 con 0x55d3f2008db0 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3163217524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 msgr2=0x55d3f2018e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc09aa30 tx=0x7fd6dc09a5e0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3163217524 shutdown_connections 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f20116e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f1e646b0 0x55d3f2024540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/3163217524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f2008db0 0x55d3f2018e70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3163217524 >> 172.21.15.23:0/3163217524 conn(0x55d3f1ffdf50 msgr2=0x55d3f20030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3163217524 shutdown_connections 2023-11-17T00:06:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/3163217524 wait complete. 2023-11-17T00:06:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 -- start start 2023-11-17T00:06:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.149+0000 7fd70c8fcb80 1 Processor -- start 2023-11-17T00:06:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2008db0 0x55d3f2014ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f2017d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2018240 con 0x55d3f2009b10 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f20183b0 con 0x55d3f1e646b0 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2018570 con 0x55d3f2008db0 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd70c8fcb80 10 client.0 _ll_register_callbacks cb 0x55d3f1f19720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:42550/0 (socket says 172.21.15.23:42550) 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 -- 172.21.15.23:0/4045396646 learned_addr learned my addr 172.21.15.23:0/4045396646 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2008db0 0x55d3f2014ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f2017d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 -- 172.21.15.23:0/4045396646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2008db0 msgr2=0x55d3f2014ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2008db0 0x55d3f2014ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 -- 172.21.15.23:0/4045396646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 msgr2=0x55d3f2017d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f2017d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f259c700 1 -- 172.21.15.23:0/4045396646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3f1ddb5d0 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.150+0000 7fd6f2d9d700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f2017d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6f259c700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4011b30 tx=0x7fd6e401f9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d642d42921e5bb1e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4009e60 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6dc0a2e60 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fd6e401fc90 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e4004620 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd70c8fcb80 10 client.25736 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d3f1d70c00 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.151+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc0a2e60 con 0x55d3f1e646b0 2023-11-17T00:06:30.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd6e4027600 con 0x55d3f1e646b0 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 10 client.25736 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 20 client.25736 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f2072200 0x55d3f2074610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 4 client.25736 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d3f1f28600 con 0x55d3f2072200 2023-11-17T00:06:30.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.152+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f2072200 0x55d3f2074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.153+0000 7fd6d9ffb700 10 client.25736 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:30.153+0000 7fd6ebfff700 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f2072200 0x55d3f2074610 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56310e24846c230f server_cookie=b8e04d8b388797a7 in_seq=0 out_seq=0 2023-11-17T00:06:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:06:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:06:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:06:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:06:30.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:06:30.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:29 smithi023 ceph-mon[97824]: pgmap v532: 129 pgs: 129 active+clean; 307 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 490 op/s 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:06:30.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:29 smithi173 ceph-mon[131534]: pgmap v532: 129 pgs: 129 active+clean; 307 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 490 op/s 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:06:30.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:29 smithi156 ceph-mon[131105]: pgmap v532: 129 pgs: 129 active+clean; 307 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 490 op/s 2023-11-17T00:06:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.518+0000 7fd6d9ffb700 1 -- 172.21.15.23:0/4045396646 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d3f1f28600 con 0x55d3f2072200 2023-11-17T00:06:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.518+0000 7fd6d9ffb700 10 client.25736 handle_command_reply: tid=0 2023-11-17T00:06:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.518+0000 7fd70c8fcb80 1 client.25736 shutdown 2023-11-17T00:06:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.518+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 msgr2=0x55d3f20147a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.518+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4011b30 tx=0x7fd6e401f9f0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 shutdown_connections 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x55d3f2072200 0x55d3f2074610 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3f2009b10 0x55d3f2017d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3f1e646b0 0x55d3f20147a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 --2- 172.21.15.23:0/4045396646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3f2008db0 0x55d3f2014ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 >> 172.21.15.23:0/4045396646 conn(0x55d3f1ffdf50 msgr2=0x55d3f20116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd6d9ffb700 0 client.25736 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.519+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 shutdown_connections 2023-11-17T00:06:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.520+0000 7fd70c8fcb80 1 -- 172.21.15.23:0/4045396646 wait complete. 2023-11-17T00:06:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.521+0000 7fd70c8fcb80 20 client.25736 trim_cache size 0 max 16384 2023-11-17T00:06:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:06:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:06:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:06:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:06:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 msgr2=0x7fd6e00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 0x7fd6e00a3680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4005fd0 tx=0x7fd6e4009310 comp rx=0 tx=0).stop 2023-11-17T00:06:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 msgr2=0x7fd6f4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009fd0 tx=0x7fd6dc00b790 comp rx=0 tx=0).stop 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 shutdown_connections 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd6e00a11d0 0x7fd6e00a3680 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6f4108170 0x7fd6f419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f40734d0 0x7fd6f4198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 --2- 172.21.15.23:0/1127943076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6f4074ab0 0x7fd6f4198760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 >> 172.21.15.23:0/1127943076 conn(0x7fd6f40fb8e0 msgr2=0x7fd6f40fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 shutdown_connections 2023-11-17T00:06:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:31.523+0000 7fd6f8fff700 1 -- 172.21.15.23:0/1127943076 wait complete. 2023-11-17T00:06:32.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:32 smithi023 ceph-mon[97824]: pgmap v533: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 407 op/s 2023-11-17T00:06:32.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:32 smithi173 ceph-mon[131534]: pgmap v533: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 407 op/s 2023-11-17T00:06:32.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:32 smithi156 ceph-mon[131105]: pgmap v533: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 407 op/s 2023-11-17T00:06:33.329 INFO:tasks.workunit.client.0.smithi023.stdout:cleaned up hierarchy 2023-11-17T00:06:33.860 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:06:33.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:06:33.977 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-11-17T00:06:33.977 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-11-17T00:06:34.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:34 smithi023 ceph-mon[97824]: pgmap v534: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 402 op/s 2023-11-17T00:06:34.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:34 smithi173 ceph-mon[131534]: pgmap v534: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 402 op/s 2023-11-17T00:06:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:34 smithi156 ceph-mon[131105]: pgmap v534: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 402 op/s 2023-11-17T00:06:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:36 smithi023 ceph-mon[97824]: pgmap v535: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 394 op/s 2023-11-17T00:06:36.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:36 smithi173 ceph-mon[131534]: pgmap v535: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 394 op/s 2023-11-17T00:06:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:36 smithi156 ceph-mon[131105]: pgmap v535: 129 pgs: 129 active+clean; 313 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 394 op/s 2023-11-17T00:06:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:38 smithi156 ceph-mon[131105]: pgmap v536: 129 pgs: 129 active+clean; 314 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2023-11-17T00:06:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:38 smithi173 ceph-mon[131534]: pgmap v536: 129 pgs: 129 active+clean; 314 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2023-11-17T00:06:38.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:38 smithi023 ceph-mon[97824]: pgmap v536: 129 pgs: 129 active+clean; 314 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2023-11-17T00:06:40.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:39 smithi023 ceph-mon[97824]: pgmap v537: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2023-11-17T00:06:40.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:39 smithi156 ceph-mon[131105]: pgmap v537: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2023-11-17T00:06:40.409 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:39 smithi173 ceph-mon[131534]: pgmap v537: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2023-11-17T00:06:42.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:42 smithi023 ceph-mon[97824]: pgmap v538: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 302 op/s 2023-11-17T00:06:42.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:42 smithi173 ceph-mon[131534]: pgmap v538: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 302 op/s 2023-11-17T00:06:42.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:42 smithi156 ceph-mon[131105]: pgmap v538: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 302 op/s 2023-11-17T00:06:44.161 INFO:tasks.workunit.client.0.smithi023.stdout:ok 2023-11-17T00:06:44.167 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:06:44.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:06:44.266 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-11-17T00:06:44.267 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-11-17T00:06:44.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:44 smithi023 ceph-mon[97824]: pgmap v539: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 528 KiB/s wr, 267 op/s 2023-11-17T00:06:44.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:44 smithi173 ceph-mon[131534]: pgmap v539: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 528 KiB/s wr, 267 op/s 2023-11-17T00:06:44.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:44 smithi156 ceph-mon[131105]: pgmap v539: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 528 KiB/s wr, 267 op/s 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 -- 172.21.15.23:0/2641529293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1002c0 msgr2=0x7f0b1c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2641529293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c1006d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009990 tx=0x7f0b0c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 -- 172.21.15.23:0/2641529293 shutdown_connections 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2641529293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c101e40 0x7f0b1c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2641529293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c1006d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2641529293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 -- 172.21.15.23:0/2641529293 >> 172.21.15.23:0/2641529293 conn(0x7f0b1c0fb8e0 msgr2=0x7f0b1c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 -- 172.21.15.23:0/2641529293 shutdown_connections 2023-11-17T00:06:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.697+0000 7f0b211ff700 1 -- 172.21.15.23:0/2641529293 wait complete. 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.699+0000 7f0b211ff700 1 Processor -- start 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.699+0000 7f0b211ff700 1 -- start start 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c101e40 0x7f0b1c19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c19dba0 con 0x7f0b1c1014b0 2023-11-17T00:06:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c19dce0 con 0x7f0b1c1002c0 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b211ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c19de70 con 0x7f0b1c101e40 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59126/0 (socket says 172.21.15.23:59126) 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 -- 172.21.15.23:0/2176845520 learned_addr learned my addr 172.21.15.23:0/2176845520 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c101e40 0x7f0b1c19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 -- 172.21.15.23:0/2176845520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c101e40 msgr2=0x7f0b1c19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c101e40 0x7f0b1c19d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 -- 172.21.15.23:0/2176845520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c1002c0 msgr2=0x7f0b1c196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.700+0000 7f0b1a59c700 1 -- 172.21.15.23:0/2176845520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0095f0 con 0x7f0b1c1014b0 2023-11-17T00:06:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000d950 tx=0x7f0b1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b03fff700 1 -- 172.21.15.23:0/2176845520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000f890 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b03fff700 1 -- 172.21.15.23:0/2176845520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b10010460 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c10bc00 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b03fff700 1 -- 172.21.15.23:0/2176845520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000e610 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.701+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c10ba70 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.704+0000 7f0b03fff700 1 -- 172.21.15.23:0/2176845520 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0b100108d0 con 0x7f0b1c1014b0 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.705+0000 7f0b34ae5b80 1 Processor -- start 2023-11-17T00:06:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.705+0000 7f0b34ae5b80 1 -- start start 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.705+0000 7f0b03fff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 0x7f0b040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x5626487649e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x562648765390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648665000 con 0x562648764f20 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626486b7840 con 0x5626487641c0 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626486b0cd0 con 0x5626485bb6b0 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b03fff700 1 -- 172.21.15.23:0/2176845520 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0b100cca00 con 0x7f0b1c1014b0 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:52518/0 (socket says 172.21.15.23:52518) 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/979227436 learned_addr learned my addr 172.21.15.23:0/979227436 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x5626487649e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 0x7f0b040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x562648765390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/979227436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 msgr2=0x5626487649e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x5626487649e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/979227436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 msgr2=0x562648765390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x562648765390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.706+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/979227436 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626485325d0 con 0x5626485bb6b0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.707+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c015000 tx=0x7f0b0c009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d5f4287bc953318 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.707+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 0x7f0b040a3680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b140078d0 tx=0x7f0b1400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.707+0000 7f0b017fa700 1 -- 172.21.15.23:0/979227436 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c01d070 con 0x5626485bb6b0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.707+0000 7f0b017fa700 1 -- 172.21.15.23:0/979227436 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b0c01e040 con 0x5626485bb6b0 2023-11-17T00:06:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.707+0000 7f0b017fa700 1 -- 172.21.15.23:0/979227436 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c004500 con 0x5626485bb6b0 2023-11-17T00:06:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/979227436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 msgr2=0x562648763c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c015000 tx=0x7f0b0c009ee0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/979227436 shutdown_connections 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x562648765390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x562648763c80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/979227436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x5626487649e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/979227436 >> 172.21.15.23:0/979227436 conn(0x562648754f50 msgr2=0x56264875a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/979227436 shutdown_connections 2023-11-17T00:06:44.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/979227436 wait complete. 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.708+0000 7f0b34ae5b80 1 -- start start 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 Processor -- start 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x56264876d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562648764f20 0x562648774580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648652880 con 0x5626485bb6b0 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648652520 con 0x5626487641c0 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626485348c0 con 0x562648764f20 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b34ae5b80 10 client.0 _ll_register_callbacks cb 0x562648670720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b1ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b1ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59138/0 (socket says 172.21.15.23:59138) 2023-11-17T00:06:44.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.709+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/1508654016 learned_addr learned my addr 172.21.15.23:0/1508654016 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x56264876d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562648764f20 0x562648774580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/1508654016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562648764f20 msgr2=0x562648774580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562648764f20 0x562648774580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 msgr2=0x56264876d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x56264876d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 -- 172.21.15.23:0/1508654016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c109030 con 0x5626485bb6b0 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x56264876d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00b5c0 tx=0x7f0b0c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b5a0da6dd1ac26 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.710+0000 7f0afeff5700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c01d040 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0b0c005980 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562648752710 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b0c00b6f0 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 10 client.15690 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562648677130 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562648752710 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0b0c004ba0 con 0x5626485bb6b0 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 10 client.15690 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 20 client.15690 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487789d0 0x56264877ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 4 client.15690 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.711+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562648673040 con 0x5626487789d0 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.712+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487789d0 0x56264877ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.712+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487789d0 0x56264877ade0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6856814c6d6419b server_cookie=37842d242f4f764c in_seq=0 out_seq=0 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.712+0000 7f0afeff5700 10 client.15690 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0afeff5700 1 -- 172.21.15.23:0/1508654016 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562648673040 con 0x5626487789d0 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0afeff5700 10 client.15690 handle_command_reply: tid=0 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0b34ae5b80 1 client.15690 shutdown 2023-11-17T00:06:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 msgr2=0x56264876d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00b5c0 tx=0x7f0b0c004060 comp rx=0 tx=0).stop 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.713+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 shutdown_connections 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562648764f20 0x562648774580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487789d0 0x56264877ade0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5626485bb6b0 0x56264876d000 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/1508654016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x56264876d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0afeff5700 0 client.15690 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 >> 172.21.15.23:0/1508654016 conn(0x562648754f50 msgr2=0x5626487597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 shutdown_connections 2023-11-17T00:06:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/1508654016 wait complete. 2023-11-17T00:06:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.714+0000 7f0b34ae5b80 20 client.15690 trim_cache size 0 max 16384 2023-11-17T00:06:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.729+0000 7f0b34ae5b80 1 Processor -- start 2023-11-17T00:06:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.729+0000 7f0b34ae5b80 1 -- start start 2023-11-17T00:06:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x56264877f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x562648774510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264876cbd0 con 0x562648764f20 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264875b5c0 con 0x5626487641c0 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648754bf0 con 0x5626485bb6b0 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b1b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b1b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59146/0 (socket says 172.21.15.23:59146) 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b1b59e700 1 -- 172.21.15.23:0/3331334609 learned_addr learned my addr 172.21.15.23:0/3331334609 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x562648774510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.730+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x56264877f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 -- 172.21.15.23:0/3331334609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 msgr2=0x56264877f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x56264877f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 -- 172.21.15.23:0/3331334609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 msgr2=0x562648774510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x562648774510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 -- 172.21.15.23:0/3331334609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562648676e80 con 0x562648764f20 2023-11-17T00:06:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0b14017780 tx=0x7f0b14009130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f52daa313ff5e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b01ffb700 1 -- 172.21.15.23:0/3331334609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14029070 con 0x562648764f20 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b01ffb700 1 -- 172.21.15.23:0/3331334609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b14020040 con 0x562648764f20 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.731+0000 7f0b01ffb700 1 -- 172.21.15.23:0/3331334609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1401f050 con 0x562648764f20 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/3331334609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 msgr2=0x56264876b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0b14017780 tx=0x7f0b14009130 comp rx=0 tx=0).stop 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/3331334609 shutdown_connections 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x56264876b1c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626485bb6b0 0x56264877f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/3331334609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626487641c0 0x562648774510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/3331334609 >> 172.21.15.23:0/3331334609 conn(0x562648754f50 msgr2=0x56264875a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/3331334609 shutdown_connections 2023-11-17T00:06:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/3331334609 wait complete. 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.732+0000 7f0b34ae5b80 1 -- start start 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 Processor -- start 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626485bb6b0 0x56264876fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626487641c0 0x562648770290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626487737f0 con 0x562648764f20 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648773960 con 0x5626485bb6b0 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x562648773b20 con 0x5626487641c0 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b34ae5b80 10 client.0 _ll_register_callbacks cb 0x7f0b10095e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b1b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b1b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59154/0 (socket says 172.21.15.23:59154) 2023-11-17T00:06:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b1b59e700 1 -- 172.21.15.23:0/561033315 learned_addr learned my addr 172.21.15.23:0/561033315 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626485bb6b0 0x56264876fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.733+0000 7f0b1a59c700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626487641c0 0x562648770290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 -- 172.21.15.23:0/561033315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626487641c0 msgr2=0x562648770290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626487641c0 0x562648770290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 -- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626485bb6b0 msgr2=0x56264876fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626485bb6b0 0x56264876fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 -- 172.21.15.23:0/561033315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b14011040 con 0x562648764f20 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b1b59e700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009250 tx=0x7f0b14007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b532b5ef53c4ecc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0afeff5700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14032070 con 0x562648764f20 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f0b14025770 con 0x562648764f20 2023-11-17T00:06:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b0c01e040 con 0x562648764f20 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b1401f020 con 0x562648764f20 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b34ae5b80 10 client.15696 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.734+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5626484c7c00 con 0x562648764f20 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b0c01e040 con 0x562648764f20 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0b14035020 con 0x562648764f20 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 10 client.15696 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 20 client.15696 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487cd2c0 0x5626487cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 4 client.15696 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56264867f600 con 0x5626487cd2c0 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.735+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487cd2c0 0x5626487cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.736+0000 7f0afeff5700 10 client.15696 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.736+0000 7f0b1ad9d700 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487cd2c0 0x5626487cf6d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e98b5bcdd269d72 server_cookie=5d20c2932f0b1f43 in_seq=0 out_seq=0 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0afeff5700 1 -- 172.21.15.23:0/561033315 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56264867f600 con 0x5626487cd2c0 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0afeff5700 10 client.15696 handle_command_reply: tid=0 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 client.15696 shutdown 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 msgr2=0x5626487732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009250 tx=0x7f0b14007660 comp rx=0 tx=0).stop 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 shutdown_connections 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x5626487cd2c0 0x5626487cf6d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562648764f20 0x5626487732b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626485bb6b0 0x56264876fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0afeff5700 0 client.15696 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 --2- 172.21.15.23:0/561033315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626487641c0 0x562648770290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 >> 172.21.15.23:0/561033315 conn(0x562648754f50 msgr2=0x56264876b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.761+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 shutdown_connections 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b34ae5b80 1 -- 172.21.15.23:0/561033315 wait complete. 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b34ae5b80 20 client.15696 trim_cache size 0 max 16384 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:06:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:06:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 msgr2=0x7f0b040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 0x7f0b040a3680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0b140078d0 tx=0x7f0b1400b040 comp rx=0 tx=0).stop 2023-11-17T00:06:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 msgr2=0x7f0b1c196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000d950 tx=0x7f0b1000dc60 comp rx=0 tx=0).stop 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 shutdown_connections 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0b040a11d0 0x7f0b040a3680 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b1c101e40 0x7f0b1c19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b1c1002c0 0x7f0b1c196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 --2- 172.21.15.23:0/2176845520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b1c1014b0 0x7f0b1c196560 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.762+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 >> 172.21.15.23:0/2176845520 conn(0x7f0b1c0fb8e0 msgr2=0x7f0b1c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.763+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 shutdown_connections 2023-11-17T00:06:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:44.763+0000 7f0b211ff700 1 -- 172.21.15.23:0/2176845520 wait complete. 2023-11-17T00:06:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:46 smithi173 ceph-mon[131534]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 514 KiB/s wr, 179 op/s 2023-11-17T00:06:46.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:46 smithi156 ceph-mon[131105]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 514 KiB/s wr, 179 op/s 2023-11-17T00:06:46.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:46 smithi023 ceph-mon[97824]: pgmap v540: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 514 KiB/s wr, 179 op/s 2023-11-17T00:06:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:48 smithi173 ceph-mon[131534]: pgmap v541: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 144 KiB/s wr, 79 op/s 2023-11-17T00:06:48.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:48 smithi156 ceph-mon[131105]: pgmap v541: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 144 KiB/s wr, 79 op/s 2023-11-17T00:06:48.776 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:48 smithi023 ceph-mon[97824]: pgmap v541: 129 pgs: 129 active+clean; 315 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 144 KiB/s wr, 79 op/s 2023-11-17T00:06:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:49 smithi023 ceph-mon[97824]: pgmap v542: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 57 op/s 2023-11-17T00:06:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:49 smithi173 ceph-mon[131534]: pgmap v542: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 57 op/s 2023-11-17T00:06:50.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:49 smithi156 ceph-mon[131105]: pgmap v542: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 57 op/s 2023-11-17T00:06:52.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:52 smithi023 ceph-mon[97824]: pgmap v543: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 29 op/s 2023-11-17T00:06:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:52 smithi173 ceph-mon[131534]: pgmap v543: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 29 op/s 2023-11-17T00:06:52.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:52 smithi156 ceph-mon[131105]: pgmap v543: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 29 op/s 2023-11-17T00:06:54.364 INFO:tasks.workunit.client.0.smithi023.stdout:ok 2023-11-17T00:06:54.369 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-17T00:06:54.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-17T00:06:54.460 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-11-17T00:06:56.008 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-11-17T00:06:56.013 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:54 smithi156 ceph-mon[131105]: pgmap v544: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.6 KiB/s wr, 17 op/s 2023-11-17T00:06:56.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:54 smithi173 ceph-mon[131534]: pgmap v544: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.6 KiB/s wr, 17 op/s 2023-11-17T00:06:56.014 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:54 smithi023 ceph-mon[97824]: pgmap v544: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.6 KiB/s wr, 17 op/s 2023-11-17T00:06:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:56 smithi173 ceph-mon[131534]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2023-11-17T00:06:56.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:56 smithi156 ceph-mon[131105]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2023-11-17T00:06:56.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:56 smithi023 ceph-mon[97824]: pgmap v545: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.5 KiB/s wr, 18 op/s 2023-11-17T00:06:57.432 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p dir 2023-11-17T00:06:57.437 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-17T00:06:57.438 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-17T00:06:57.440 INFO:tasks.workunit.client.0.smithi023.stdout:ceph.dir.pin="-1" 2023-11-17T00:06:57.441 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin dir 2023-11-17T00:06:57.442 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-17T00:06:57.444 INFO:tasks.workunit.client.0.smithi023.stdout:setfattr: dir: Invalid argument 2023-11-17T00:06:57.444 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin -v 1 dir 2023-11-17T00:06:57.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-17T00:06:57.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'ceph.dir.pin="1"' 2023-11-17T00:06:57.451 INFO:tasks.workunit.client.0.smithi023.stdout:ceph.dir.pin="1" 2023-11-17T00:06:57.451 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -x ceph.dir.pin dir 2023-11-17T00:06:57.455 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-17T00:06:57.455 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-17T00:06:57.458 INFO:tasks.workunit.client.0.smithi023.stdout:ceph.dir.pin="-1" 2023-11-17T00:06:57.459 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin.distributed dir 2023-11-17T00:06:57.459 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-17T00:06:57.461 INFO:tasks.workunit.client.0.smithi023.stdout:setfattr: dir: Invalid argument 2023-11-17T00:06:57.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 dir 2023-11-17T00:06:57.465 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -x ceph.dir.pin.distributed dir 2023-11-17T00:06:57.469 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin.random dir 2023-11-17T00:06:57.469 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-17T00:06:57.471 INFO:tasks.workunit.client.0.smithi023.stdout:setfattr: dir: Invalid argument 2023-11-17T00:06:57.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.pin.random -v 0.01 dir 2023-11-17T00:06:57.475 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -x ceph.dir.pin.random dir 2023-11-17T00:06:57.478 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.subvolume dir 2023-11-17T00:06:57.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-17T00:06:57.480 INFO:tasks.workunit.client.0.smithi023.stdout:setfattr: dir: Invalid argument 2023-11-17T00:06:57.481 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n ceph.dir.subvolume -v 1 dir 2023-11-17T00:06:57.488 INFO:tasks.workunit.client.0.smithi023.stderr:setfattr: dir: Invalid argument 2023-11-17T00:06:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:06:57.489 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-11-17T00:06:57.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 -- 172.21.15.23:0/3962311820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 msgr2=0x7f3eb01030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/3962311820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb01030a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00098f0 tx=0x7f3ea0009c00 comp rx=0 tx=0).stop 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 -- 172.21.15.23:0/3962311820 shutdown_connections 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/3962311820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb0105f00 0x7f3eb0108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/3962311820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb01030a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/3962311820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb01035e0 0x7f3eb01059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 -- 172.21.15.23:0/3962311820 >> 172.21.15.23:0/3962311820 conn(0x7f3eb00fa720 msgr2=0x7f3eb00fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 -- 172.21.15.23:0/3962311820 shutdown_connections 2023-11-17T00:06:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.912+0000 7f3eb61ff700 1 -- 172.21.15.23:0/3962311820 wait complete. 2023-11-17T00:06:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.913+0000 7f3eb61ff700 1 Processor -- start 2023-11-17T00:06:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.913+0000 7f3eb61ff700 1 -- start start 2023-11-17T00:06:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb01035e0 0x7f3eb0196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 0x7f3eb019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb019dbc0 con 0x7f3eb0100cc0 2023-11-17T00:06:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb019dd00 con 0x7f3eb0105f00 2023-11-17T00:06:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eb61ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb019de90 con 0x7f3eb01035e0 2023-11-17T00:06:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56232/0 (socket says 172.21.15.23:56232) 2023-11-17T00:06:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/737544515 learned_addr learned my addr 172.21.15.23:0/737544515 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb01035e0 0x7f3eb0196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.914+0000 7f3eaffff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 0x7f3eb019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/737544515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb01035e0 msgr2=0x7f3eb0196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb01035e0 0x7f3eb0196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/737544515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 msgr2=0x7f3eb019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 0x7f3eb019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/737544515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4009520 con 0x7f3eb0100cc0 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaffff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 0x7f3eb019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009fd0 tx=0x7f3ea000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eacff9700 1 -- 172.21.15.23:0/737544515 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea001c070 con 0x7f3eb0100cc0 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eacff9700 1 -- 172.21.15.23:0/737544515 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ea000f7d0 con 0x7f3eb0100cc0 2023-11-17T00:06:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea0009520 con 0x7f3eb0100cc0 2023-11-17T00:06:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eacff9700 1 -- 172.21.15.23:0/737544515 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004b90 con 0x7f3eb0100cc0 2023-11-17T00:06:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.915+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb010bc10 con 0x7f3eb0100cc0 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.919+0000 7f3eacff9700 1 -- 172.21.15.23:0/737544515 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ea0020070 con 0x7f3eb0100cc0 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 Processor -- start 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 -- start start 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6c9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f5c9000 con 0x563d2f6c8f20 2023-11-17T00:06:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f61b840 con 0x563d2f51f6b0 2023-11-17T00:06:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f614cd0 con 0x563d2f6c81c0 2023-11-17T00:06:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.920+0000 7f3eacff9700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 0x7f3e980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eacff9700 1 -- 172.21.15.23:0/737544515 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f3ea00cd9d0 con 0x7f3eb0100cc0 2023-11-17T00:06:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:59858/0 (socket says 172.21.15.23:59858) 2023-11-17T00:06:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 -- 172.21.15.23:0/3713715457 learned_addr learned my addr 172.21.15.23:0/3713715457 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 0x7f3e980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 -- 172.21.15.23:0/3713715457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 msgr2=0x563d2f6c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 -- 172.21.15.23:0/3713715457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 msgr2=0x563d2f6c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaffff700 1 -- 172.21.15.23:0/3713715457 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d2f4965d0 con 0x563d2f51f6b0 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.921+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.922+0000 7f3eaffff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 0x7f3e980a3600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8019c30 tx=0x7f3ea800f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.922+0000 7f3eaffff700 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80195d0 tx=0x7f3ea80198e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d84940f404367b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.922+0000 7f3e95ffb700 1 -- 172.21.15.23:0/3713715457 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8026070 con 0x563d2f51f6b0 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.922+0000 7f3e95ffb700 1 -- 172.21.15.23:0/3713715457 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ea801e070 con 0x563d2f51f6b0 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.922+0000 7f3e95ffb700 1 -- 172.21.15.23:0/3713715457 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800e040 con 0x563d2f51f6b0 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/3713715457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 msgr2=0x563d2f6c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80195d0 tx=0x7f3ea80198e0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/3713715457 shutdown_connections 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6c7c80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/3713715457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/3713715457 >> 172.21.15.23:0/3713715457 conn(0x563d2f6b8f50 msgr2=0x563d2f6be100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/3713715457 shutdown_connections 2023-11-17T00:06:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.923+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/3713715457 wait complete. 2023-11-17T00:06:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 -- start start 2023-11-17T00:06:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 Processor -- start 2023-11-17T00:06:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6d85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f5b6880 con 0x563d2f6c8f20 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f5b6520 con 0x563d2f51f6b0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.924+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f4988c0 con 0x563d2f6c81c0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3ec9b03b80 10 client.0 _ll_register_callbacks cb 0x563d2f5d4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6d85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.23:59868/0 (socket says 172.21.15.23:59868) 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/4114470063 learned_addr learned my addr 172.21.15.23:0/4114470063 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 -- 172.21.15.23:0/4114470063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 msgr2=0x563d2f6d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 -- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 msgr2=0x563d2f6d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 -- 172.21.15.23:0/4114470063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea8012d40 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3eaffff700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6d85f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80195d0 tx=0x7f3ea801ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e294ff26bda62f2e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.925+0000 7f3e8effd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea802e070 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f3ea8009300 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d2f6b6710 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ea800e070 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3ec9b03b80 10 client.15708 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563d2f5db130 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.926+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563d2f6b6710 con 0x563d2f6c8f20 2023-11-17T00:06:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3ea8031020 con 0x563d2f6c8f20 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 20 client.15708 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f6dce40 0x563d2f6df250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 4 client.15708 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d2f5d7040 con 0x563d2f6dce40 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.927+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f6dce40 0x563d2f6df250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.928+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f6dce40 0x563d2f6df250 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2503d2b127ec236e server_cookie=a2be173d50d9a2c1 in_seq=0 out_seq=0 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.928+0000 7f3e8effd700 10 client.15708 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.928+0000 7f3e8effd700 1 -- 172.21.15.23:0/4114470063 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563d2f5d7040 con 0x563d2f6dce40 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.928+0000 7f3e8effd700 10 client.15708 handle_command_reply: tid=0 2023-11-17T00:06:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.928+0000 7f3ec9b03b80 1 client.15708 shutdown 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 msgr2=0x563d2f6d85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6d85f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80195d0 tx=0x7f3ea801ae00 comp rx=0 tx=0).stop 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 shutdown_connections 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f6dce40 0x563d2f6df250 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f6c8f20 0x563d2f6d85f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f51f6b0 0x563d2f6d1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/4114470063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 >> 172.21.15.23:0/4114470063 conn(0x563d2f6b8f50 msgr2=0x563d2f6bd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3e8effd700 0 client.15708 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 shutdown_connections 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/4114470063 wait complete. 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.929+0000 7f3ec9b03b80 20 client.15708 trim_cache size 0 max 16384 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.944+0000 7f3ec9b03b80 1 Processor -- start 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 -- start start 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f6cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6d0be0 con 0x563d2f51f6b0 2023-11-17T00:06:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6bef60 con 0x563d2f6c8f20 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6cf070 con 0x563d2f6c81c0 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56252/0 (socket says 172.21.15.23:56252) 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/603129016 learned_addr learned my addr 172.21.15.23:0/603129016 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3eaffff700 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f6cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.945+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/603129016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 msgr2=0x563d2f6d85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/603129016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 msgr2=0x563d2f6cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f6cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/603129016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d2f5dae80 con 0x563d2f51f6b0 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0029000 tx=0x7f3ea002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8e1be0a55afe97 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3e967fc700 1 -- 172.21.15.23:0/603129016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000ba30 con 0x563d2f51f6b0 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3e967fc700 1 -- 172.21.15.23:0/603129016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ea009aac0 con 0x563d2f51f6b0 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.946+0000 7f3e967fc700 1 -- 172.21.15.23:0/603129016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00cb230 con 0x563d2f51f6b0 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/603129016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 msgr2=0x563d2f6bf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0029000 tx=0x7f3ea002e040 comp rx=0 tx=0).stop 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/603129016 shutdown_connections 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f6cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6bf1a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/603129016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/603129016 >> 172.21.15.23:0/603129016 conn(0x563d2f6b8f50 msgr2=0x563d2f6be0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/603129016 shutdown_connections 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/603129016 wait complete. 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.947+0000 7f3ec9b03b80 1 -- start start 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 Processor -- start 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f72ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6d8640 con 0x563d2f51f6b0 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6d87b0 con 0x563d2f6c8f20 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d2f6d8970 con 0x563d2f6c81c0 2023-11-17T00:06:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3ec9b03b80 10 client.0 _ll_register_callbacks cb 0x563d2f5d4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3eaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56264/0 (socket says 172.21.15.23:56264) 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/2574185831 learned_addr learned my addr 172.21.15.23:0/2574185831 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3eaffff700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f72ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.948+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/2574185831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 msgr2=0x563d2f6d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 msgr2=0x563d2f72ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f72ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 -- 172.21.15.23:0/2574185831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb010bc10 con 0x563d2f51f6b0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3eaf7fe700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0098d10 tx=0x7f3ea00cb0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efde0845f49226e server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3e8effd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-11-17T00:06:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00b3240 con 0x563d2f51f6b0 2023-11-17T00:06:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f3ea00a2e80 con 0x563d2f51f6b0 2023-11-17T00:06:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ea801e070 con 0x563d2f51f6b0 2023-11-17T00:06:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.949+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ea00cb1a0 con 0x563d2f51f6b0 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 10 client.15714 fetch_fsmap learned FSMap version 33 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563d2f42bc00 con 0x563d2f51f6b0 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ea801e070 con 0x563d2f51f6b0 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3ea00bd8a0 con 0x563d2f51f6b0 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 10 client.15714 fetch_fsmap finished waiting for FSMap version 33 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 20 client.15714 populate_metadata read hostname 'smithi023' 2023-11-17T00:06:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f7306d0 0x563d2f732ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 4 client.15714 mds_command: new command op to 24572 tid=0[{"prefix": "flush journal"}] 2023-11-17T00:06:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 --> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563d2f5e3600 con 0x563d2f7306d0 2023-11-17T00:06:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.950+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f7306d0 0x563d2f732ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.951+0000 7f3e8effd700 10 client.15714 ms_handle_connect on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.951+0000 7f3eaeffd700 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f7306d0 0x563d2f732ae0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4463cd40d91aa7 server_cookie=a5307b0e0efe72ae in_seq=0 out_seq=0 2023-11-17T00:06:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3e8effd700 1 -- 172.21.15.23:0/2574185831 <== mds.0 v2:172.21.15.156:6838/1637342449 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563d2f5e3600 con 0x563d2f7306d0 2023-11-17T00:06:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3e8effd700 10 client.15714 handle_command_reply: tid=0 2023-11-17T00:06:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 client.15714 shutdown 2023-11-17T00:06:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 msgr2=0x563d2f6d3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0098d10 tx=0x7f3ea00cb0c0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 shutdown_connections 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563d2f6c8f20 0x563d2f72ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.156:6838/1637342449,v1:172.21.15.156:6839/1637342449] conn(0x563d2f7306d0 0x563d2f732ae0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d2f51f6b0 0x563d2f6d3780 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 --2- 172.21.15.23:0/2574185831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d2f6c81c0 0x563d2f6d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3e8effd700 0 client.15714 ms_handle_reset on v2:172.21.15.156:6838/1637342449 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 >> 172.21.15.23:0/2574185831 conn(0x563d2f6b8f50 msgr2=0x563d2f6e3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 shutdown_connections 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.964+0000 7f3ec9b03b80 1 -- 172.21.15.23:0/2574185831 wait complete. 2023-11-17T00:06:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3ec9b03b80 20 client.15714 trim_cache size 0 max 16384 2023-11-17T00:06:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2023-11-17T00:06:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2023-11-17T00:06:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2023-11-17T00:06:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2023-11-17T00:06:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 msgr2=0x7f3e980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 0x7f3e980a3600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8019c30 tx=0x7f3ea800f000 comp rx=0 tx=0).stop 2023-11-17T00:06:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 msgr2=0x7f3eb0196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009fd0 tx=0x7f3ea000b790 comp rx=0 tx=0).stop 2023-11-17T00:06:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 shutdown_connections 2023-11-17T00:06:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3eb0105f00 0x7f3eb019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3eb0100cc0 0x7f3eb0196040 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f3e980a1150 0x7f3e980a3600 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 --2- 172.21.15.23:0/737544515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3eb01035e0 0x7f3eb0196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.965+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 >> 172.21.15.23:0/737544515 conn(0x7f3eb00fa720 msgr2=0x7f3eb0101920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.966+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 shutdown_connections 2023-11-17T00:06:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2023-11-17T00:06:57.966+0000 7f3eb61ff700 1 -- 172.21.15.23:0/737544515 wait complete. 2023-11-17T00:06:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:58 smithi173 ceph-mon[131534]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 14 op/s 2023-11-17T00:06:58.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:58 smithi156 ceph-mon[131105]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 14 op/s 2023-11-17T00:06:58.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:58 smithi023 ceph-mon[97824]: pgmap v546: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.1 KiB/s wr, 14 op/s 2023-11-17T00:06:58.925 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-17T00:06:59.004 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=30ea35db2aab44929baa459c1f2ed37c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_1c9565146c8647d4e00eb6c7ae626dc192680e7b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-17T00:06:59.007 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-11-17T00:06:59.143 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:06:59.161 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-11-17T00:06:59.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:06:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 -- 172.21.15.23:0/3533954004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 msgr2=0x7faa5c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 --2- 172.21.15.23:0/3533954004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1006a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009990 tx=0x7faa4c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 -- 172.21.15.23:0/3533954004 shutdown_connections 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 --2- 172.21.15.23:0/3533954004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 0x7faa5c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 --2- 172.21.15.23:0/3533954004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1006a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 --2- 172.21.15.23:0/3533954004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 0x7faa5c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 -- 172.21.15.23:0/3533954004 >> 172.21.15.23:0/3533954004 conn(0x7faa5c0fb870 msgr2=0x7faa5c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:59.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 -- 172.21.15.23:0/3533954004 shutdown_connections 2023-11-17T00:06:59.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.407+0000 7faa60dff700 1 -- 172.21.15.23:0/3533954004 wait complete. 2023-11-17T00:06:59.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 Processor -- start 2023-11-17T00:06:59.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 -- start start 2023-11-17T00:06:59.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 0x7faa5c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 0x7faa5c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c19fd40 con 0x7faa5c100290 2023-11-17T00:06:59.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c19fe80 con 0x7faa5c101480 2023-11-17T00:06:59.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.408+0000 7faa60dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c1a0010 con 0x7faa5c101e10 2023-11-17T00:06:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56284/0 (socket says 172.21.15.23:56284) 2023-11-17T00:06:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 -- 172.21.15.23:0/555757013 learned_addr learned my addr 172.21.15.23:0/555757013 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5ad9d700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 0x7faa5c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa53fff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 0x7faa5c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 -- 172.21.15.23:0/555757013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 msgr2=0x7faa5c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 0x7faa5c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 -- 172.21.15.23:0/555757013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 msgr2=0x7faa5c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 0x7faa5c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 -- 172.21.15.23:0/555757013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c0095f0 con 0x7faa5c100290 2023-11-17T00:06:59.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.409+0000 7faa5a59c700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004730 tx=0x7faa4c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.410+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c01d070 con 0x7faa5c100290 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.410+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa4c021410 con 0x7faa5c100290 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.410+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c10bc40 con 0x7faa5c100290 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.410+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c017650 con 0x7faa5c100290 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.410+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c10bab0 con 0x7faa5c100290 2023-11-17T00:06:59.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.411+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c1928d0 con 0x7faa5c100290 2023-11-17T00:06:59.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.413+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa4c021580 con 0x7faa5c100290 2023-11-17T00:06:59.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.414+0000 7faa537fe700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 0x7faa480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.414+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7faa4c0ce0a0 con 0x7faa5c100290 2023-11-17T00:06:59.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.415+0000 7faa53fff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 0x7faa480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.415+0000 7faa53fff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 0x7faa480a3600 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faa44009c60 tx=0x7faa44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:59.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.419+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa4c0974d0 con 0x7faa5c100290 2023-11-17T00:06:59.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.629+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa5c02cfe0 con 0x7faa5c100290 2023-11-17T00:06:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.631+0000 7faa537fe700 1 -- 172.21.15.23:0/555757013 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24603 (secure 0 0 0) 0x7faa4c09aaf0 con 0x7faa5c100290 2023-11-17T00:06:59.633 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:06:59.634 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:05:51.321983+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.633+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 msgr2=0x7faa480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.633+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 0x7faa480a3600 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faa44009c60 tx=0x7faa44009260 comp rx=0 tx=0).stop 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.633+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 msgr2=0x7faa5c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.633+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004730 tx=0x7faa4c004760 comp rx=0 tx=0).stop 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.633+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 shutdown_connections 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faa480a1150 0x7faa480a3600 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa5c101e10 0x7faa5c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa5c100290 0x7faa5c1981c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 --2- 172.21.15.23:0/555757013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa5c101480 0x7faa5c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 >> 172.21.15.23:0/555757013 conn(0x7faa5c0fb870 msgr2=0x7faa5c106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:59.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 shutdown_connections 2023-11-17T00:06:59.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.634+0000 7faa60dff700 1 -- 172.21.15.23:0/555757013 wait complete. 2023-11-17T00:06:59.655 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:06:59.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:06:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 -- 172.21.15.23:0/705059603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 msgr2=0x7fab10101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 --2- 172.21.15.23:0/705059603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10101900 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009990 tx=0x7faaf8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 -- 172.21.15.23:0/705059603 shutdown_connections 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 --2- 172.21.15.23:0/705059603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab10101e40 0x7fab10108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 --2- 172.21.15.23:0/705059603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab101002c0 0x7fab101006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 --2- 172.21.15.23:0/705059603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10101900 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 -- 172.21.15.23:0/705059603 >> 172.21.15.23:0/705059603 conn(0x7fab100fb8a0 msgr2=0x7fab100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:06:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 -- 172.21.15.23:0/705059603 shutdown_connections 2023-11-17T00:06:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.864+0000 7fab157ff700 1 -- 172.21.15.23:0/705059603 wait complete. 2023-11-17T00:06:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.865+0000 7fab157ff700 1 Processor -- start 2023-11-17T00:06:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.865+0000 7fab157ff700 1 -- start start 2023-11-17T00:06:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.865+0000 7fab157ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab101002c0 0x7fab10198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.865+0000 7fab157ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab157ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 0x7fab1019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab157ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1019fdd0 con 0x7fab101014b0 2023-11-17T00:06:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab157ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1019ff10 con 0x7fab101002c0 2023-11-17T00:06:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab157ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab101a00a0 con 0x7fab10101e40 2023-11-17T00:06:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 0x7fab1019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56308/0 (socket says 172.21.15.23:56308) 2023-11-17T00:06:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 0x7fab1019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:39076/0 (socket says 172.21.15.23:39076) 2023-11-17T00:06:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 -- 172.21.15.23:0/3631543718 learned_addr learned my addr 172.21.15.23:0/3631543718 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:06:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab0effd700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab101002c0 0x7fab10198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 -- 172.21.15.23:0/3631543718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 msgr2=0x7fab1019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 0x7fab1019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 -- 172.21.15.23:0/3631543718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab101002c0 msgr2=0x7fab10198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:06:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab101002c0 0x7fab10198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:06:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 -- 172.21.15.23:0/3631543718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf80095f0 con 0x7fab101014b0 2023-11-17T00:06:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.866+0000 7fab07fff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004760 tx=0x7faaf8004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.867+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf801d070 con 0x7fab101014b0 2023-11-17T00:06:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.867+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaf8004830 con 0x7fab101014b0 2023-11-17T00:06:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.867+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1010bc70 con 0x7fab101014b0 2023-11-17T00:06:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.867+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1010bae0 con 0x7fab101014b0 2023-11-17T00:06:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.867+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf8005370 con 0x7fab101014b0 2023-11-17T00:06:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.868+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab101929d0 con 0x7fab101014b0 2023-11-17T00:06:59.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.870+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faaf80054d0 con 0x7fab101014b0 2023-11-17T00:06:59.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.871+0000 7fab0cff9700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 0x7faafc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:06:59.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.871+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7faaf80ce690 con 0x7fab101014b0 2023-11-17T00:06:59.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.872+0000 7fab0effd700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 0x7faafc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:06:59.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.872+0000 7fab0effd700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 0x7faafc0a3600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fab10101310 tx=0x7fab00009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:06:59.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:06:59.874+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faaf8097ac0 con 0x7fab101014b0 2023-11-17T00:07:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:59 smithi023 ceph-mon[97824]: pgmap v547: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2023-11-17T00:07:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:06:59 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/555757013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:00.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.089+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab10061700 con 0x7fab101014b0 2023-11-17T00:07:00.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.089+0000 7fab0cff9700 1 -- 172.21.15.23:0/3631543718 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7faaf809b0e0 con 0x7fab101014b0 2023-11-17T00:07:00.092 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:07:00.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 msgr2=0x7faafc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 0x7faafc0a3600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fab10101310 tx=0x7fab00009260 comp rx=0 tx=0).stop 2023-11-17T00:07:00.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 msgr2=0x7fab10198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004760 tx=0x7faaf8004790 comp rx=0 tx=0).stop 2023-11-17T00:07:00.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 shutdown_connections 2023-11-17T00:07:00.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7faafc0a1150 0x7faafc0a3600 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab10101e40 0x7fab1019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab101002c0 0x7fab10198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 --2- 172.21.15.23:0/3631543718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab101014b0 0x7fab10198790 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 >> 172.21.15.23:0/3631543718 conn(0x7fab100fb8a0 msgr2=0x7fab10106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:00.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 shutdown_connections 2023-11-17T00:07:00.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.092+0000 7fab157ff700 1 -- 172.21.15.23:0/3631543718 wait complete. 2023-11-17T00:07:00.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:07:00.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 -- 172.21.15.23:0/1176376401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 msgr2=0x7f6c58072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/1176376401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c58072f80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6c540098f0 tx=0x7f6c54009c00 comp rx=0 tx=0).stop 2023-11-17T00:07:00.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 -- 172.21.15.23:0/1176376401 shutdown_connections 2023-11-17T00:07:00.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/1176376401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 0x7f6c5810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/1176376401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c58072f80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/1176376401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c580739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 -- 172.21.15.23:0/1176376401 >> 172.21.15.23:0/1176376401 conn(0x7f6c580fb8a0 msgr2=0x7f6c580fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:00.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.324+0000 7f6c5ffff700 1 -- 172.21.15.23:0/1176376401 shutdown_connections 2023-11-17T00:07:00.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.325+0000 7f6c5ffff700 1 -- 172.21.15.23:0/1176376401 wait complete. 2023-11-17T00:07:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 Processor -- start 2023-11-17T00:07:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 -- start start 2023-11-17T00:07:00.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c5819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 0x7f6c581a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581a4290 con 0x7f6c58074b30 2023-11-17T00:07:00.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581a43d0 con 0x7f6c58073550 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5ffff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581a4560 con 0x7f6c5810c890 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5d59a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c5819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5d59a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56326/0 (socket says 172.21.15.23:56326) 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.326+0000 7f6c5d59a700 1 -- 172.21.15.23:0/2027622686 learned_addr learned my addr 172.21.15.23:0/2027622686 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5e59c700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 0x7f6c581a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 -- 172.21.15.23:0/2027622686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 msgr2=0x7f6c581a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 0x7f6c581a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 -- 172.21.15.23:0/2027622686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 msgr2=0x7f6c5819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c5819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 -- 172.21.15.23:0/2027622686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480095f0 con 0x7f6c58074b30 2023-11-17T00:07:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5dd9b700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c5819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5d59a700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800ebf0 tx=0x7f6c4800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:00.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48009a10 con 0x7f6c58074b30 2023-11-17T00:07:00.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c4800cc90 con 0x7f6c58074b30 2023-11-17T00:07:00.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c54009520 con 0x7f6c58074b30 2023-11-17T00:07:00.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.327+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48018750 con 0x7f6c58074b30 2023-11-17T00:07:00.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.328+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c58112280 con 0x7f6c58074b30 2023-11-17T00:07:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.329+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c581023e0 con 0x7f6c58074b30 2023-11-17T00:07:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.331+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6c480188b0 con 0x7f6c58074b30 2023-11-17T00:07:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.332+0000 7f6c4effd700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 0x7f6c440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.332+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f6c48014070 con 0x7f6c58074b30 2023-11-17T00:07:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.332+0000 7f6c5dd9b700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 0x7f6c440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.333+0000 7f6c5dd9b700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 0x7f6c440a3680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009fd0 tx=0x7f6c54019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.334+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6c48096cf0 con 0x7f6c58074b30 2023-11-17T00:07:00.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:59 smithi173 ceph-mon[131534]: pgmap v547: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2023-11-17T00:07:00.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:06:59 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/555757013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:00.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:59 smithi156 ceph-mon[131105]: pgmap v547: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2023-11-17T00:07:00.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:06:59 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/555757013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:00.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.550+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6c58067b00 con 0x7f6c58074b30 2023-11-17T00:07:00.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.550+0000 7f6c4effd700 1 -- 172.21.15.23:0/2027622686 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6c4809a310 con 0x7f6c58074b30 2023-11-17T00:07:00.553 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:07:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.552+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 msgr2=0x7f6c440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.552+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 0x7f6c440a3680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009fd0 tx=0x7f6c54019040 comp rx=0 tx=0).stop 2023-11-17T00:07:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.552+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 msgr2=0x7f6c5819cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.552+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800ebf0 tx=0x7f6c4800ef00 comp rx=0 tx=0).stop 2023-11-17T00:07:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 shutdown_connections 2023-11-17T00:07:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f6c440a11d0 0x7f6c440a3680 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c5810c890 0x7f6c581a3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c58073550 0x7f6c5819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 --2- 172.21.15.23:0/2027622686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c58074b30 0x7f6c5819cc50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 >> 172.21.15.23:0/2027622686 conn(0x7f6c580fb8a0 msgr2=0x7f6c580fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:00.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 shutdown_connections 2023-11-17T00:07:00.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.553+0000 7f6c5ffff700 1 -- 172.21.15.23:0/2027622686 wait complete. 2023-11-17T00:07:00.575 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:07:00.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 -- 172.21.15.23:0/865613632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 msgr2=0x7f41541083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 --2- 172.21.15.23:0/865613632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f41541083e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f414c00b0a0 tx=0x7f414c00b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 -- 172.21.15.23:0/865613632 shutdown_connections 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 --2- 172.21.15.23:0/865613632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f41541083e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 --2- 172.21.15.23:0/865613632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154100d40 0x7f4154103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 --2- 172.21.15.23:0/865613632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154103660 0x7f4154105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 -- 172.21.15.23:0/865613632 >> 172.21.15.23:0/865613632 conn(0x7f41540fa7c0 msgr2=0x7f41540fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:00.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 -- 172.21.15.23:0/865613632 shutdown_connections 2023-11-17T00:07:00.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.783+0000 7f41595ff700 1 -- 172.21.15.23:0/865613632 wait complete. 2023-11-17T00:07:00.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.784+0000 7f41595ff700 1 Processor -- start 2023-11-17T00:07:00.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.784+0000 7f41595ff700 1 -- start start 2023-11-17T00:07:00.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 0x7f4154198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154103660 0x7f41541988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415419fee0 con 0x7f4154105f80 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541a0020 con 0x7f4154100d40 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41595ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541a01b0 con 0x7f4154103660 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56344/0 (socket says 172.21.15.23:56344) 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 -- 172.21.15.23:0/989947000 learned_addr learned my addr 172.21.15.23:0/989947000 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:00.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41527fc700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154103660 0x7f41541988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f4152ffd700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 0x7f4154198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 -- 172.21.15.23:0/989947000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154103660 msgr2=0x7f41541988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154103660 0x7f41541988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 -- 172.21.15.23:0/989947000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 msgr2=0x7f4154198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 0x7f4154198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:00.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f41537fe700 1 -- 172.21.15.23:0/989947000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c009d00 con 0x7f4154105f80 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.785+0000 7f4152ffd700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 0x7f4154198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f41537fe700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f414c009fd0 tx=0x7f414c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c00e040 con 0x7f4154105f80 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f414c0094c0 con 0x7f4154105f80 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415410bc70 con 0x7f4154105f80 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c005260 con 0x7f4154105f80 2023-11-17T00:07:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.786+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415410bae0 con 0x7f4154105f80 2023-11-17T00:07:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.787+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415410be00 con 0x7f4154105f80 2023-11-17T00:07:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.789+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f414c003a00 con 0x7f4154105f80 2023-11-17T00:07:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.791+0000 7f413bfff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 0x7f413c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:00.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.791+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f414c0ce370 con 0x7f4154105f80 2023-11-17T00:07:00.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.791+0000 7f4152ffd700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 0x7f413c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:00.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.791+0000 7f4152ffd700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 0x7f413c0a3600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4144009cb0 tx=0x7f4144009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:00.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:00.793+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f414c0977a0 con 0x7f4154105f80 2023-11-17T00:07:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.002+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4154067b00 con 0x7f4154105f80 2023-11-17T00:07:01.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.002+0000 7f413bfff700 1 -- 172.21.15.23:0/989947000 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4154067b00 con 0x7f4154105f80 2023-11-17T00:07:01.014 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:07:01.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.004+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 msgr2=0x7f413c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.004+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 0x7f413c0a3600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4144009cb0 tx=0x7f4144009260 comp rx=0 tx=0).stop 2023-11-17T00:07:01.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 msgr2=0x7f415419f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f414c009fd0 tx=0x7f414c00bce0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 shutdown_connections 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154105f80 0x7f415419f910 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4154100d40 0x7f4154198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f413c0a1150 0x7f413c0a3600 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 --2- 172.21.15.23:0/989947000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4154103660 0x7f41541988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 >> 172.21.15.23:0/989947000 conn(0x7f41540fa7c0 msgr2=0x7f41540ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 shutdown_connections 2023-11-17T00:07:01.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.005+0000 7f41595ff700 1 -- 172.21.15.23:0/989947000 wait complete. 2023-11-17T00:07:01.026 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:07:01.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 -- 172.21.15.23:0/3532235055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 msgr2=0x7f9278105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3532235055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f9278105a40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9264009990 tx=0x7f9264009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 -- 172.21.15.23:0/3532235055 shutdown_connections 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3532235055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278105f80 0x7f92781083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3532235055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278100d40 0x7f9278103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3532235055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f9278105a40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 -- 172.21.15.23:0/3532235055 >> 172.21.15.23:0/3532235055 conn(0x7f92780fa7c0 msgr2=0x7f92780fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 -- 172.21.15.23:0/3532235055 shutdown_connections 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.236+0000 7f927f1ff700 1 -- 172.21.15.23:0/3532235055 wait complete. 2023-11-17T00:07:01.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.237+0000 7f927f1ff700 1 Processor -- start 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 -- start start 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278100d40 0x7f9278198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278105f80 0x7f927819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819fe10 con 0x7f9278103660 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819ff50 con 0x7f9278105f80 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927f1ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781a00e0 con 0x7f9278100d40 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f9277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f9277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60180/0 (socket says 172.21.15.23:60180) 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f9277fff700 1 -- 172.21.15.23:0/3007970575 learned_addr learned my addr 172.21.15.23:0/3007970575 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927d79c700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278105f80 0x7f927819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.238+0000 7f927cf9b700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278100d40 0x7f9278198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 -- 172.21.15.23:0/3007970575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278100d40 msgr2=0x7f9278198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278100d40 0x7f9278198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 -- 172.21.15.23:0/3007970575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278105f80 msgr2=0x7f927819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278105f80 0x7f927819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 -- 172.21.15.23:0/3007970575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640095f0 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9277fff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f926400b680 tx=0x7f9264005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926401d070 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9264021410 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264017650 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927810bcd0 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.239+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927810bb40 con 0x7f9278103660 2023-11-17T00:07:01.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.241+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9278107b70 con 0x7f9278103660 2023-11-17T00:07:01.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.242+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9264021580 con 0x7f9278103660 2023-11-17T00:07:01.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.244+0000 7f9275ffb700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.244+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f92640ce090 con 0x7f9278103660 2023-11-17T00:07:01.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.244+0000 7f927cf9b700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.245+0000 7f927cf9b700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 0x7f92680a3680 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f926c009c60 tx=0x7f926c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:01.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.248+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f92640974c0 con 0x7f9278103660 2023-11-17T00:07:01.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:00 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3631543718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:00 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2027622686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:00 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3631543718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:00 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2027622686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:00 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3631543718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:00 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2027622686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.459+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9278061a80 con 0x7f9278103660 2023-11-17T00:07:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.460+0000 7f9275ffb700 1 -- 172.21.15.23:0/3007970575 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f926409aae0 con 0x7f9278103660 2023-11-17T00:07:01.462 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:07:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 msgr2=0x7f92680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 0x7f92680a3680 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f926c009c60 tx=0x7f926c009260 comp rx=0 tx=0).stop 2023-11-17T00:07:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 msgr2=0x7f92781987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f926400b680 tx=0x7f9264005740 comp rx=0 tx=0).stop 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 shutdown_connections 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9278105f80 0x7f927819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9278100d40 0x7f9278198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 --2- 172.21.15.23:0/3007970575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9278103660 0x7f92781987d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 >> 172.21.15.23:0/3007970575 conn(0x7f92780fa7c0 msgr2=0x7f92780ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 shutdown_connections 2023-11-17T00:07:01.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.462+0000 7f927f1ff700 1 -- 172.21.15.23:0/3007970575 wait complete. 2023-11-17T00:07:01.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-11-17T00:07:01.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 -- 172.21.15.23:0/1789524195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681002b0 msgr2=0x7f88681006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1789524195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681002b0 0x7f88681006c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8858009990 tx=0x7f8858009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 -- 172.21.15.23:0/1789524195 shutdown_connections 2023-11-17T00:07:01.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1789524195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8868101e30 0x7f88681086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1789524195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681002b0 0x7f88681006c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1789524195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88681014a0 0x7f88681018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 -- 172.21.15.23:0/1789524195 >> 172.21.15.23:0/1789524195 conn(0x7f88680fb890 msgr2=0x7f88680fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 -- 172.21.15.23:0/1789524195 shutdown_connections 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.685+0000 7f886d1ff700 1 -- 172.21.15.23:0/1789524195 wait complete. 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.686+0000 7f886d1ff700 1 Processor -- start 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.686+0000 7f886d1ff700 1 -- start start 2023-11-17T00:07:01.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 0x7f8868198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 0x7f886819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886819fdb0 con 0x7f88681014a0 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886819fef0 con 0x7f88681002b0 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886d1ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88681a0080 con 0x7f8868101e30 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60208/0 (socket says 172.21.15.23:60208) 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 -- 172.21.15.23:0/1298589024 learned_addr learned my addr 172.21.15.23:0/1298589024 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f8866d9d700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 0x7f8868198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886759e700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 0x7f886819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 -- 172.21.15.23:0/1298589024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 msgr2=0x7f886819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 0x7f886819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 -- 172.21.15.23:0/1298589024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 msgr2=0x7f8868198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 0x7f8868198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886659c700 1 -- 172.21.15.23:0/1298589024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c009520 con 0x7f88681014a0 2023-11-17T00:07:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f886759e700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 0x7f886819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.687+0000 7f8866d9d700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 0x7f8868198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f886659c700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f885c00eb30 tx=0x7f885c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c009880 con 0x7f88681014a0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f885c0044e0 con 0x7f88681014a0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580095f0 con 0x7f88681014a0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c003d80 con 0x7f88681014a0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.688+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886810bc80 con 0x7f88681014a0 2023-11-17T00:07:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.689+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8868192910 con 0x7f88681014a0 2023-11-17T00:07:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.691+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f885c003f20 con 0x7f88681014a0 2023-11-17T00:07:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.694+0000 7f884ffff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 0x7f88500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.694+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f885c014070 con 0x7f88681014a0 2023-11-17T00:07:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.694+0000 7f8866d9d700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 0x7f88500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:01.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.694+0000 7f8866d9d700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 0x7f88500a3600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f885800b2e0 tx=0x7f8858005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:01.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.697+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f885c096590 con 0x7f88681014a0 2023-11-17T00:07:01.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.972+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f886802cfe0 con 0x7f88681014a0 2023-11-17T00:07:01.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.984+0000 7f884ffff700 1 -- 172.21.15.23:0/1298589024 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+733239 (secure 0 0 0) 0x7f885c094e60 con 0x7f88681014a0 2023-11-17T00:07:01.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:07:01.996 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:07:01.996 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 18, 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 14150, 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2023-11-17T00:07:01.997 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 499313593 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 499313593 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:01.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/499313593", 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2023-11-16T23:51:30.957922+0000", 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540138322906710015, 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 24115, 2023-11-17T00:07:01.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540138322906710015, 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.000 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.002 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.003 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.012 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2023-11-17T00:07:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2023-11-17T00:07:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2023-11-17T00:07:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2023-11-17T00:07:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2023-11-17T00:07:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2023-11-17T00:07:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2023-11-17T00:07:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2023-11-17T00:07:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2023-11-17T00:07:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2023-11-17T00:07:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2023-11-17T00:07:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2023-11-17T00:07:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-17T00:07:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2023-11-17T00:07:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-17T00:07:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-17T00:07:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-17T00:07:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2023-11-17T00:07:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2023-11-17T00:07:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2023-11-17T00:07:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2023-11-17T00:07:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_password": { 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_password", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web password", 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_user": { 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_user", 2023-11-17T00:07:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web user", 2023-11-17T00:07:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2023-11-17T00:07:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2023-11-17T00:07:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-17T00:07:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2023-11-17T00:07:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2023-11-17T00:07:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2023-11-17T00:07:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-17T00:07:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2023-11-17T00:07:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2023-11-17T00:07:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2023-11-17T00:07:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2023-11-17T00:07:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2023-11-17T00:07:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-17T00:07:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2023-11-17T00:07:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2023-11-17T00:07:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2023-11-17T00:07:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2023-11-17T00:07:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2023-11-17T00:07:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2023-11-17T00:07:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-17T00:07:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2023-11-17T00:07:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2023-11-17T00:07:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2023-11-17T00:07:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-17T00:07:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2023-11-17T00:07:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2023-11-17T00:07:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2023-11-17T00:07:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-17T00:07:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2023-11-17T00:07:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2023-11-17T00:07:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2023-11-17T00:07:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-17T00:07:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2023-11-17T00:07:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-17T00:07:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2023-11-17T00:07:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2023-11-17T00:07:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-17T00:07:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-17T00:07:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2023-11-17T00:07:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2023-11-17T00:07:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2023-11-17T00:07:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-17T00:07:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2023-11-17T00:07:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2023-11-17T00:07:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2023-11-17T00:07:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-17T00:07:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2023-11-17T00:07:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2023-11-17T00:07:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2023-11-17T00:07:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-17T00:07:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2023-11-17T00:07:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2023-11-17T00:07:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2023-11-17T00:07:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2023-11-17T00:07:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2023-11-17T00:07:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2023-11-17T00:07:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2023-11-17T00:07:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2023-11-17T00:07:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2023-11-17T00:07:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-17T00:07:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2023-11-17T00:07:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2023-11-17T00:07:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2023-11-17T00:07:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2023-11-17T00:07:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2023-11-17T00:07:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-17T00:07:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-17T00:07:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-17T00:07:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-17T00:07:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-17T00:07:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-17T00:07:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-17T00:07:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2023-11-17T00:07:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-17T00:07:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2023-11-17T00:07:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2023-11-17T00:07:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2023-11-17T00:07:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2023-11-17T00:07:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-17T00:07:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2023-11-17T00:07:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2023-11-17T00:07:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2023-11-17T00:07:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2023-11-17T00:07:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2023-11-17T00:07:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-17T00:07:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_password": { 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_password", 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web password", 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_user": { 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_user", 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web user", 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-17T00:07:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2023-11-17T00:07:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2023-11-17T00:07:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2023-11-17T00:07:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2023-11-17T00:07:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-17T00:07:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2023-11-17T00:07:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2023-11-17T00:07:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-17T00:07:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2023-11-17T00:07:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-17T00:07:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-17T00:07:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-17T00:07:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-17T00:07:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-17T00:07:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-17T00:07:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-17T00:07:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-17T00:07:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-17T00:07:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-17T00:07:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-17T00:07:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-17T00:07:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-17T00:07:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-17T00:07:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-17T00:07:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-17T00:07:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2023-11-17T00:07:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2023-11-17T00:07:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2023-11-17T00:07:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2023-11-17T00:07:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2023-11-17T00:07:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2023-11-17T00:07:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2023-11-17T00:07:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2023-11-17T00:07:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2023-11-17T00:07:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2023-11-17T00:07:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2023-11-17T00:07:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2023-11-17T00:07:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2023-11-17T00:07:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2023-11-17T00:07:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-17T00:07:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2023-11-17T00:07:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2023-11-17T00:07:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2023-11-17T00:07:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.871 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.872 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.872 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.872 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2023-11-17T00:07:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2023-11-17T00:07:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "noautoscale": { 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "noautoscale", 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "global autoscale flag", 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2023-11-17T00:07:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2023-11-17T00:07:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2023-11-17T00:07:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2023-11-17T00:07:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2023-11-17T00:07:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "drive_group_interval": { 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "drive_group_interval", 2023-11-17T00:07:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300.0", 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2023-11-17T00:07:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-17T00:07:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2023-11-17T00:07:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2023-11-17T00:07:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2023-11-17T00:07:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2023-11-17T00:07:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2023-11-17T00:07:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2023-11-17T00:07:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2023-11-17T00:07:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-17T00:07:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2023-11-17T00:07:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2023-11-17T00:07:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-17T00:07:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2023-11-17T00:07:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2023-11-17T00:07:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-17T00:07:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2023-11-17T00:07:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.268 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.276 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.280 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.285 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.290 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:03.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2023-11-17T00:07:03.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2023-11-17T00:07:03.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.309 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.314 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-17T00:07:03.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.322 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 24119, 2023-11-17T00:07:03.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540138322906710015, 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.324 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.327 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.332 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2023-11-17T00:07:03.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2023-11-17T00:07:03.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2023-11-17T00:07:03.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2023-11-17T00:07:03.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2023-11-17T00:07:03.352 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.354 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2023-11-17T00:07:03.355 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2023-11-17T00:07:03.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2023-11-17T00:07:03.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2023-11-17T00:07:03.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.361 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2023-11-17T00:07:03.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.364 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.365 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:03.366 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.369 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.370 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.371 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.374 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2023-11-17T00:07:03.380 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2023-11-17T00:07:03.381 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.382 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.389 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.390 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2023-11-17T00:07:03.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.393 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2023-11-17T00:07:03.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.395 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_password": { 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_password", 2023-11-17T00:07:03.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web password", 2023-11-17T00:07:03.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_user": { 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_user", 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web user", 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:03.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.403 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.405 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2023-11-17T00:07:03.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-17T00:07:03.410 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.412 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-17T00:07:03.413 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2023-11-17T00:07:03.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2023-11-17T00:07:03.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.426 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-17T00:07:03.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.429 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.430 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2023-11-17T00:07:03.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2023-11-17T00:07:03.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2023-11-17T00:07:03.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:03.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2023-11-17T00:07:03.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2023-11-17T00:07:03.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.453 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.454 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2023-11-17T00:07:03.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2023-11-17T00:07:03.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.462 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.465 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-17T00:07:03.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.470 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.473 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2023-11-17T00:07:03.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_password": { 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_password", 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web password", 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.480 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_user": { 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_user", 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web user", 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2023-11-17T00:07:03.482 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.483 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-17T00:07:03.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.498 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.503 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.504 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2023-11-17T00:07:03.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2023-11-17T00:07:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-17T00:07:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2023-11-17T00:07:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-17T00:07:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2023-11-17T00:07:03.542 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-17T00:07:03.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-17T00:07:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2023-11-17T00:07:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-17T00:07:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-17T00:07:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-17T00:07:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.595 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.596 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-17T00:07:03.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.598 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.599 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:03.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.604 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.604 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.604 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-17T00:07:03.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.611 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-17T00:07:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.613 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.614 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.618 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2023-11-17T00:07:03.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.620 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.624 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.627 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.629 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.635 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:03.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.643 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:03.643 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:03.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.643 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2023-11-17T00:07:03.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.645 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.645 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2023-11-17T00:07:03.645 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:03.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2023-11-17T00:07:03.650 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.654 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.655 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.656 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.658 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.659 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.659 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.659 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.659 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.659 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.661 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2023-11-17T00:07:03.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2023-11-17T00:07:03.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-17T00:07:03.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.668 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.672 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.673 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.674 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2023-11-17T00:07:03.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.676 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.679 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.680 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.681 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.683 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.684 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2023-11-17T00:07:03.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.689 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.689 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.689 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2023-11-17T00:07:03.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.693 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2023-11-17T00:07:03.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-17T00:07:03.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.705 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.709 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2023-11-17T00:07:03.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.716 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2023-11-17T00:07:03.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-17T00:07:03.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.723 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.724 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.730 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.733 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.734 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.736 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.740 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.745 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2023-11-17T00:07:03.746 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.749 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.754 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.755 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.757 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2023-11-17T00:07:03.758 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.758 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.758 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.758 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2023-11-17T00:07:03.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2023-11-17T00:07:03.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.771 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.772 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.772 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.772 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2023-11-17T00:07:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.824 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2023-11-17T00:07:03.828 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.829 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.831 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.836 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.839 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.841 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.850 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.855 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "noautoscale": { 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "noautoscale", 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.858 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "global autoscale flag", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-17T00:07:03.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:03.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2023-11-17T00:07:03.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.865 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2023-11-17T00:07:03.865 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2023-11-17T00:07:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.872 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.875 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.876 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2023-11-17T00:07:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2023-11-17T00:07:03.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.882 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2023-11-17T00:07:03.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-17T00:07:03.886 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.887 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.888 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.892 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.894 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2023-11-17T00:07:03.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.897 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.898 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.901 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:03.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-17T00:07:03.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.905 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2023-11-17T00:07:03.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.907 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:03.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.911 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.918 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:03.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.922 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.923 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.924 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.926 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.929 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.930 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.932 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.935 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.942 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2023-11-17T00:07:03.950 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "drive_group_interval": { 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "drive_group_interval", 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300.0", 2023-11-17T00:07:03.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.963 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.966 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.969 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2023-11-17T00:07:03.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2023-11-17T00:07:03.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:03.979 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2023-11-17T00:07:03.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.986 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2023-11-17T00:07:03.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.990 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:03.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.992 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2023-11-17T00:07:03.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2023-11-17T00:07:03.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:03.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:03.998 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.002 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.003 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.006 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.007 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.008 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.011 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.013 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.014 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.016 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.021 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2023-11-17T00:07:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.029 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.034 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2023-11-17T00:07:04.037 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.038 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.038 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.038 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2023-11-17T00:07:04.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2023-11-17T00:07:04.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2023-11-17T00:07:04.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-17T00:07:04.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.057 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-17T00:07:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.068 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.073 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2023-11-17T00:07:04.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2023-11-17T00:07:04.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.079 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.080 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.085 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2023-11-17T00:07:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.089 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.095 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.100 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.105 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2023-11-17T00:07:04.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2023-11-17T00:07:04.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.119 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.123 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2023-11-17T00:07:04.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: "restful" 2023-11-17T00:07:04.133 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.137 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.141 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.146 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2023-11-17T00:07:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2023-11-17T00:07:04.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2023-11-17T00:07:04.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.159 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2023-11-17T00:07:04.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:04.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2023-11-17T00:07:04.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.168 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2023-11-17T00:07:04.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2023-11-17T00:07:04.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-17T00:07:04.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.175 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.179 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2023-11-17T00:07:04.185 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.186 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:04.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-17T00:07:04.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2023-11-17T00:07:04.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_password": { 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_password", 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web password", 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "alertmanager_web_user": { 2023-11-17T00:07:04.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alertmanager_web_user", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Alertmanager web user", 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2023-11-17T00:07:04.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.213 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2023-11-17T00:07:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2023-11-17T00:07:04.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-17T00:07:04.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.220 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2023-11-17T00:07:04.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2023-11-17T00:07:04.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-17T00:07:04.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.227 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2023-11-17T00:07:04.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2023-11-17T00:07:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2023-11-17T00:07:04.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2023-11-17T00:07:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.244 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.249 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.252 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.255 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.258 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.259 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-17T00:07:04.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-17T00:07:04.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2023-11-17T00:07:04.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2023-11-17T00:07:04.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2023-11-17T00:07:04.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_password": { 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_password", 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web password", 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_web_user": { 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_web_user", 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus web user", 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2023-11-17T00:07:04.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.283 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.288 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.293 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.295 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.303 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2023-11-17T00:07:04.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.309 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-17T00:07:04.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2023-11-17T00:07:04.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-17T00:07:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-17T00:07:04.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.335 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.338 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.340 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.343 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.345 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-17T00:07:04.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.349 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-17T00:07:04.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.356 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-17T00:07:04.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.361 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-17T00:07:04.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-17T00:07:04.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-17T00:07:04.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2023-11-17T00:07:04.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.372 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.378 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-17T00:07:04.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2023-11-17T00:07:04.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:04.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.392 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.394 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.395 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.400 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.401 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2023-11-17T00:07:04.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.404 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:04.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2023-11-17T00:07:04.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.419 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.420 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.421 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.422 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.425 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2023-11-17T00:07:04.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2023-11-17T00:07:04.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.433 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2023-11-17T00:07:04.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.438 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.439 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.442 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.444 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.446 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.449 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.450 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.452 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2023-11-17T00:07:04.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2023-11-17T00:07:04.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-17T00:07:04.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-17T00:07:04.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.463 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.464 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.467 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.469 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2023-11-17T00:07:04.470 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2023-11-17T00:07:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:04.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2023-11-17T00:07:04.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2023-11-17T00:07:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.479 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2023-11-17T00:07:04.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2023-11-17T00:07:04.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-17T00:07:04.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.483 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.489 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.490 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.491 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.494 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.498 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2023-11-17T00:07:04.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-17T00:07:04.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-17T00:07:04.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.517 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.518 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.519 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.520 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.521 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.522 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.523 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.524 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2023-11-17T00:07:04.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2023-11-17T00:07:04.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.540 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2023-11-17T00:07:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2023-11-17T00:07:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2023-11-17T00:07:04.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2023-11-17T00:07:04.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.556 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.562 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.565 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.567 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.571 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.572 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2023-11-17T00:07:04.574 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.576 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.580 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.581 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.583 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.587 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2023-11-17T00:07:04.595 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.596 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.598 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.599 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.602 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.607 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.611 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.614 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.616 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.622 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "noautoscale": { 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "noautoscale", 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "global autoscale flag", 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.625 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-17T00:07:04.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:04.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.634 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.638 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2023-11-17T00:07:04.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.644 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.645 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2023-11-17T00:07:04.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.649 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.650 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.656 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.657 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.661 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:04.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-17T00:07:04.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2023-11-17T00:07:04.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.671 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.673 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.680 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.681 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.682 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.685 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.685 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.685 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.685 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2023-11-17T00:07:04.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2023-11-17T00:07:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.690 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.693 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.696 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.697 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.700 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.704 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.706 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.710 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.714 INFO:teuthology.orchestra.run.smithi023.stdout: "drive_group_interval": { 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "drive_group_interval", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300.0", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.717 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.721 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2023-11-17T00:07:04.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.726 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.728 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.730 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.733 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.734 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.735 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2023-11-17T00:07:04.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.739 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2023-11-17T00:07:04.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.745 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2023-11-17T00:07:04.746 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2023-11-17T00:07:04.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.757 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2023-11-17T00:07:04.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.762 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.764 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.764 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.764 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.765 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.765 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.767 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.768 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.768 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.768 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.769 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.769 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.769 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.769 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.770 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.770 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.770 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.771 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.772 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.772 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2023-11-17T00:07:04.774 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.780 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.789 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.792 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.793 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.798 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.799 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.805 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2023-11-17T00:07:04.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-17T00:07:04.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.810 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2023-11-17T00:07:04.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2023-11-17T00:07:04.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.817 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.818 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.821 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.829 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.832 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.833 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.836 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2023-11-17T00:07:04.838 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.839 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.841 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.842 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.848 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.849 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2023-11-17T00:07:04.851 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.853 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.854 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.857 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2023-11-17T00:07:04.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-17T00:07:04.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2023-11-17T00:07:04.864 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2023-11-17T00:07:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.868 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.871 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2023-11-17T00:07:04.875 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2023-11-17T00:07:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2023-11-17T00:07:04.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2023-11-17T00:07:04.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2023-11-17T00:07:04.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2023-11-17T00:07:04.884 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2023-11-17T00:07:04.885 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2023-11-17T00:07:04.886 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2023-11-17T00:07:04.887 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2023-11-17T00:07:04.889 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2023-11-17T00:07:04.889 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2023-11-17T00:07:04.889 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2023-11-17T00:07:04.889 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2023-11-17T00:07:04.890 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 3, 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 4223790019 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1775702876 2023-11-17T00:07:04.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-17T00:07:04.894 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1032636545 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-11-17T00:07:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 19269155 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2023-11-17T00:07:04.896 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:07:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:01 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/989947000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:01 smithi156 ceph-mon[131105]: pgmap v548: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.7 KiB/s wr, 14 op/s 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:01 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3007970575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:02 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1298589024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:03 smithi156 ceph-mon[131105]: pgmap v549: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:01 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/989947000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:01 smithi173 ceph-mon[131534]: pgmap v548: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.7 KiB/s wr, 14 op/s 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:01 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3007970575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:02 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1298589024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-17T00:07:04.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:03 smithi173 ceph-mon[131534]: pgmap v549: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2023-11-17T00:07:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 msgr2=0x7f88500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 0x7f88500a3600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f885800b2e0 tx=0x7f8858005fb0 comp rx=0 tx=0).stop 2023-11-17T00:07:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 msgr2=0x7f8868198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f885c00eb30 tx=0x7f885c00ee40 comp rx=0 tx=0).stop 2023-11-17T00:07:04.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 shutdown_connections 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f88500a1150 0x7f88500a3600 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8868101e30 0x7f886819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88681002b0 0x7f8868198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 --2- 172.21.15.23:0/1298589024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88681014a0 0x7f8868198770 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.991+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 >> 172.21.15.23:0/1298589024 conn(0x7f88680fb890 msgr2=0x7f8868106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.992+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 shutdown_connections 2023-11-17T00:07:04.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:01.992+0000 7f886d1ff700 1 -- 172.21.15.23:0/1298589024 wait complete. 2023-11-17T00:07:04.907 DEBUG:tasks.check_counter:Getting stats from a 2023-11-17T00:07:04.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-11-17T00:07:04.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:01 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/989947000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:01 smithi023 ceph-mon[97824]: pgmap v548: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.7 KiB/s wr, 14 op/s 2023-11-17T00:07:04.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:01 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3007970575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:04.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:02 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1298589024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-17T00:07:04.909 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:03 smithi023 ceph-mon[97824]: pgmap v549: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2023-11-17T00:07:05.998 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:07:06.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:06 smithi023 ceph-mon[97824]: pgmap v550: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2023-11-17T00:07:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:06 smithi173 ceph-mon[131534]: pgmap v550: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2023-11-17T00:07:06.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:06 smithi156 ceph-mon[131105]: pgmap v550: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2023-11-17T00:07:08.541 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:08.542 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.003951127, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000723474, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:08.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:08.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 105, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 94256, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:08.545 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012848135, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009719943, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001688983, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 103, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007981673, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000077491 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.546 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 94256 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.547 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 146, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 141, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 452496, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 79120, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.028308565, 2023-11-17T00:07:08.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.011844435, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.014953848, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003079258, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 141, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.013610891, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000096531 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:08.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:08.550 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:08.550 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:08.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 452496, 2023-11-17T00:07:08.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 79120 2023-11-17T00:07:08.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.552 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2023-11-17T00:07:08.552 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2023-11-17T00:07:08.552 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2023-11-17T00:07:08.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.568 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2023-11-17T00:07:08.568 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:08.568 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:08.568 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:08.569 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:08.571 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:08.572 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 151029, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 31, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2023-11-17T00:07:08.573 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:08.574 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2023-11-17T00:07:08.575 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 143, 2023-11-17T00:07:08.576 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 429446, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 143, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 143, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 429446, 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2023-11-17T00:07:08.577 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:08.578 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:08.578 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:08.578 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:08.578 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:08.578 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:07:08.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:08 smithi023 ceph-mon[97824]: pgmap v551: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2023-11-17T00:07:08.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:08 smithi156 ceph-mon[131105]: pgmap v551: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2023-11-17T00:07:08.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:08 smithi173 ceph-mon[131534]: pgmap v551: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2023-11-17T00:07:09.002 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-11-17T00:07:09.002 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-11-17T00:07:09.002 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-11-17T00:07:09.002 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-11-17T00:07:09.003 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-11-17T00:07:09.003 DEBUG:tasks.check_counter:Getting stats from d 2023-11-17T00:07:09.003 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-11-17T00:07:09.866 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:07:10.144 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:09 smithi023 ceph-mon[97824]: pgmap v552: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2023-11-17T00:07:10.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:09 smithi173 ceph-mon[131534]: pgmap v552: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2023-11-17T00:07:10.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:09 smithi156 ceph-mon[131105]: pgmap v552: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.5 KiB/s wr, 11 op/s 2023-11-17T00:07:11.569 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:07:11.569 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2992, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:11.570 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.004246361, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000077889, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001422100, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.001444035, 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000481345 2023-11-17T00:07:11.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2023-11-17T00:07:11.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 110, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 95136, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:11.573 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012493251, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009151492, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001856749, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 108, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007331794, 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000067886 2023-11-17T00:07:11.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 95136 2023-11-17T00:07:11.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 148, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 145, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 451200, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 81792, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:11.576 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.031058995, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012525778, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.015662857, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003727893, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 144, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.011532520, 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000080086 2023-11-17T00:07:11.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 451200, 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 81792 2023-11-17T00:07:11.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:11.579 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:11.580 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:11.581 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:11.582 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:11.583 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:11.583 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:11.583 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:11.583 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2023-11-17T00:07:11.583 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148967, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 29, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:11.584 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:11.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:11.586 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2023-11-17T00:07:11.587 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 149, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 430290, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 149, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 149, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 430290, 2023-11-17T00:07:11.588 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:11.589 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:07:12.062 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-11-17T00:07:12.062 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-11-17T00:07:12.062 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-11-17T00:07:12.062 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-11-17T00:07:12.063 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-11-17T00:07:12.063 DEBUG:tasks.check_counter:Getting stats from g 2023-11-17T00:07:12.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-11-17T00:07:12.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:12 smithi173 ceph-mon[131534]: pgmap v553: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2023-11-17T00:07:12.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:12 smithi156 ceph-mon[131105]: pgmap v553: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2023-11-17T00:07:12.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:12 smithi023 ceph-mon[97824]: pgmap v553: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2023-11-17T00:07:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:07:14.394 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:07:14.394 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:14.394 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2023-11-17T00:07:14.394 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2992, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.004295646, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000062654, 2023-11-17T00:07:14.395 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001391060, 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.001400377, 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000466792 2023-11-17T00:07:14.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.397 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:14.397 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:14.397 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:14.397 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:14.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.398 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-17T00:07:14.399 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2023-11-17T00:07:14.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.401 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:14.401 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 144, 2023-11-17T00:07:14.402 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 141, 2023-11-17T00:07:14.402 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 450048, 2023-11-17T00:07:14.402 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 79856, 2023-11-17T00:07:14.402 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:14.403 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:14.403 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.031830737, 2023-11-17T00:07:14.404 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.014295511, 2023-11-17T00:07:14.404 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.016095407, 2023-11-17T00:07:14.404 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003551824, 2023-11-17T00:07:14.404 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:14.405 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 140, 2023-11-17T00:07:14.405 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.018153775, 2023-11-17T00:07:14.405 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000129669 2023-11-17T00:07:14.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.406 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:14.406 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:14.406 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:14.407 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:14.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.407 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 450048, 2023-11-17T00:07:14.408 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 79856 2023-11-17T00:07:14.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.408 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:14.409 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2023-11-17T00:07:14.409 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 107, 2023-11-17T00:07:14.409 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2023-11-17T00:07:14.409 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 94608, 2023-11-17T00:07:14.410 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2023-11-17T00:07:14.410 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:14.410 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.013012851, 2023-11-17T00:07:14.410 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009854094, 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001716520, 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 105, 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007117717, 2023-11-17T00:07:14.411 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000067787 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:14.412 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 94608 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2023-11-17T00:07:14.413 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:14.414 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:14.415 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:14.416 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:14.416 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:14.416 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:14.416 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:14.417 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:14.418 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:14.418 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:14.419 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:14.419 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:14.419 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:14.420 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:14.420 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148967, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 29, 2023-11-17T00:07:14.421 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:14.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:14.423 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:14.423 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:14.423 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:14.424 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2023-11-17T00:07:14.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 145, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 429706, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 145, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2023-11-17T00:07:14.426 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 145, 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 429706, 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:14.427 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:14.428 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:14.428 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:07:14.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:14 smithi023 ceph-mon[97824]: pgmap v554: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2023-11-17T00:07:14.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:14 smithi156 ceph-mon[131105]: pgmap v554: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2023-11-17T00:07:14.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:14 smithi173 ceph-mon[131534]: pgmap v554: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2023-11-17T00:07:14.924 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-11-17T00:07:14.925 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-11-17T00:07:14.925 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-11-17T00:07:14.925 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-11-17T00:07:14.925 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-11-17T00:07:14.925 DEBUG:tasks.check_counter:Getting stats from j 2023-11-17T00:07:14.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-11-17T00:07:15.789 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-17T00:07:16.369 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:16 smithi023 ceph-mon[97824]: pgmap v555: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-17T00:07:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:16 smithi173 ceph-mon[131534]: pgmap v555: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-17T00:07:16.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:16 smithi156 ceph-mon[131105]: pgmap v555: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-17T00:07:17.302 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-11-17T00:07:17.302 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:17.302 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 150, 2023-11-17T00:07:17.302 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 147, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 451744, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 82976, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.032744099, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012555810, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.016188726, 2023-11-17T00:07:17.303 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003413190, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 146, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.010600197, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000072604 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:17.304 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 451744, 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 82976 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 111, 2023-11-17T00:07:17.305 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 95312, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.013454742, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.010225322, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001744920, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:17.306 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 109, 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007460166, 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000068441 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:17.307 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 95312 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2992, 2023-11-17T00:07:17.308 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001309760, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000064632, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001193463, 2023-11-17T00:07:17.309 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:17.332 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:17.332 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2023-11-17T00:07:17.332 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.001180513, 2023-11-17T00:07:17.332 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000393504 2023-11-17T00:07:17.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.333 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2023-11-17T00:07:17.334 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:17.335 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:17.336 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:17.336 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:17.336 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:17.336 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:17.337 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:17.338 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:17.338 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:17.340 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:17.340 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:17.340 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:17.340 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:17.340 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:17.341 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148967, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 29, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2023-11-17T00:07:17.342 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:17.343 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:17.343 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:17.343 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:17.344 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2023-11-17T00:07:17.345 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:17.345 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2023-11-17T00:07:17.345 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:17.411 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2023-11-17T00:07:17.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 151, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 430550, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:17.412 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 151, 2023-11-17T00:07:17.413 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2023-11-17T00:07:17.413 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2023-11-17T00:07:17.413 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 151, 2023-11-17T00:07:17.413 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 430550, 2023-11-17T00:07:17.415 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2023-11-17T00:07:17.415 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2023-11-17T00:07:17.415 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2023-11-17T00:07:17.416 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2023-11-17T00:07:17.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:17.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-11-17T00:07:17.416 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-11-17T00:07:17.788 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-11-17T00:07:17.788 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-11-17T00:07:17.788 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-11-17T00:07:17.789 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-11-17T00:07:17.789 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-11-17T00:07:17.789 DEBUG:tasks.check_counter:Getting stats from b 2023-11-17T00:07:17.789 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-11-17T00:07:18.457 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-17T00:07:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:18 smithi173 ceph-mon[131534]: pgmap v556: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-11-17T00:07:18.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:18 smithi156 ceph-mon[131105]: pgmap v556: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-11-17T00:07:18.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:18 smithi023 ceph-mon[97824]: pgmap v556: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-11-17T00:07:19.835 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 107680, 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 199136, 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 41151261, 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 214494744, 2023-11-17T00:07:19.836 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 60, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 7, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 7.898665641, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 3.391781217, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 5.510841673, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.966772298, 2023-11-17T00:07:19.837 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 199132, 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 61.422094792, 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000308449 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:19.838 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 209957, 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.034187804, 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000162 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 32042704, 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 102870608 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.839 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 34494, 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 34366, 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 60475947, 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 106194464, 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 8, 2023-11-17T00:07:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 1.987680427, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.576091794, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.592035599, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.841036190, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 34367, 2023-11-17T00:07:19.841 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 681.518648286, 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.019830612 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 15521, 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.002308529, 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000148 2023-11-17T00:07:19.842 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3152, 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 265424 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 38440, 2023-11-17T00:07:19.843 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 37840, 2023-11-17T00:07:19.844 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 40637308, 2023-11-17T00:07:19.844 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 94034903, 2023-11-17T00:07:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 12, 2023-11-17T00:07:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2023-11-17T00:07:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 2.304324220, 2023-11-17T00:07:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.671402022, 2023-11-17T00:07:19.845 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.690270621, 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.969250464, 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 37839, 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 61.085785694, 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001614360 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.846 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 16986, 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.002709545, 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000159 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 523824, 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 86720 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4216, 2023-11-17T00:07:19.848 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 51.770744223, 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.012279588 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 22429, 2023-11-17T00:07:19.849 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3.524514009, 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000157140 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 60937, 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 53360, 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2023-11-17T00:07:19.850 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 53360, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 159.727902981, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002993401 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 7577, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 348, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:19.851 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 6702, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 152, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 51, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 108, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 63, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 28, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 17, 2023-11-17T00:07:19.852 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 17, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 34377, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 3, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 3, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 4, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 74979, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 66334, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 7037, 2023-11-17T00:07:19.853 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 302, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 348, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 204, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 363, 2023-11-17T00:07:19.854 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 28075, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 197, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 246, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2023-11-17T00:07:19.855 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:19.856 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:19.856 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:19.857 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:19.857 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 7, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 332, 2023-11-17T00:07:19.858 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 39649, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 1, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 19286, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 18421, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 205, 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 24398 2023-11-17T00:07:19.859 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 1712, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 568, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 284, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 586, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 1885, 2023-11-17T00:07:19.860 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 2, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 12428, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 12426, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:19.861 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 363, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 208, 2023-11-17T00:07:19.862 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 72815, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 72804, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 72804, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 11, 2023-11-17T00:07:19.863 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 19, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 106, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 105, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 105, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 1, 2023-11-17T00:07:19.864 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 182091496, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 182114968, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1648, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 89.410033404, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.054253661 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.865 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2023-11-17T00:07:19.866 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.866 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2023-11-17T00:07:19.880 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 24, 2023-11-17T00:07:19.880 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 32099, 2023-11-17T00:07:19.880 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 32075, 2023-11-17T00:07:19.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 33, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 3830, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 3797, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 108, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 60121, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 60013, 2023-11-17T00:07:19.881 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 3, 2023-11-17T00:07:19.882 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 38371, 2023-11-17T00:07:19.882 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 38368, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 155716, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 207156 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 62430, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 121, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 60937, 2023-11-17T00:07:19.883 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 290, 2023-11-17T00:07:19.884 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 257, 2023-11-17T00:07:19.884 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2023-11-17T00:07:19.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 26953, 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 88.244672184, 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003274020 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 257, 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.347343081, 2023-11-17T00:07:19.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001351529 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2023-11-17T00:07:19.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 12630, 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 18.607763454, 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001473298 2023-11-17T00:07:19.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2023-11-17T00:07:19.892 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000323572, 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000323572 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.893 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.895 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 123, 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.120849166, 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000982513 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.897 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.897 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2023-11-17T00:07:19.897 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 942, 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 11.913941415, 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.012647496 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2023-11-17T00:07:19.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2023-11-17T00:07:19.900 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.003164119, 2023-11-17T00:07:19.900 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001582059 2023-11-17T00:07:19.900 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.900 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.902 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 112, 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.395563347, 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003531815 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:19.903 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.904 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.904 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.905 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.905 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:19.905 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4, 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.013566128, 2023-11-17T00:07:19.923 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003391532 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.924 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.925 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:19.925 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 12, 2023-11-17T00:07:19.925 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.013955415, 2023-11-17T00:07:19.925 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001162951 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2023-11-17T00:07:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 8, 2023-11-17T00:07:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.020815599, 2023-11-17T00:07:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002601949 2023-11-17T00:07:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2023-11-17T00:07:19.927 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.929 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.930 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.930 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.930 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 12316, 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 40.045945501, 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003251538 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:19.933 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:19.933 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:19.933 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:19.933 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:19.933 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 3, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2023-11-17T00:07:19.934 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 12545, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 12545, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 361 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 360, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 360, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:19.935 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:19.936 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:19.937 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:19.937 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:19.950 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:19.951 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:19.952 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:19.953 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 292671, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 130, 2023-11-17T00:07:19.954 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 616, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 7, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:19.955 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 170, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 179388, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 1501, 2023-11-17T00:07:19.956 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2023-11-17T00:07:19.957 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 81524, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 232600554, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 25, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 81524, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 81524, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 42751.934373022, 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.524409184 2023-11-17T00:07:19.958 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 81524, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 161719 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 81524, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 567, 2023-11-17T00:07:19.959 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 80957, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 5403, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 45770, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 219, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 1867, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 198, 2023-11-17T00:07:19.960 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 5, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 26307, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 69022, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2023-11-17T00:07:19.962 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 1, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2023-11-17T00:07:19.963 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2023-11-17T00:07:19.964 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2023-11-17T00:07:19.965 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2023-11-17T00:07:20.061 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2023-11-17T00:07:20.061 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 10, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 12, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 12, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2023-11-17T00:07:20.062 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 11971, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 696, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 260 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:20.063 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 61629 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:20.064 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 16, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 15, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 12426, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:20.065 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:20.066 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:20.066 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 180610, 2023-11-17T00:07:20.066 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 122052104, 2023-11-17T00:07:20.074 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:20.074 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 180610, 2023-11-17T00:07:20.074 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:20.075 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:20.075 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 180610, 2023-11-17T00:07:20.075 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 122052104, 2023-11-17T00:07:20.075 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:20.076 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:20.076 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:20.076 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:20.076 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.077 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.077 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:20.079 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 81524, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 233948540, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 81524, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 233948540, 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:20.080 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:20.081 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 81524, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 81524, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 81524, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 81524, 2023-11-17T00:07:20.082 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:20.083 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:20.083 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:20.083 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:20.084 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.084 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.084 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:20.084 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:20.084 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 12426, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1353554, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 12426, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:20.085 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 219, 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1353554, 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:20.086 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55869d1b40a0": { 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 72815, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 177920664, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:20.087 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 72815, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1648, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 177920664, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:20.088 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:20.089 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.089 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:20.089 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-17T00:07:20.247 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:19 smithi156 ceph-mon[131105]: pgmap v557: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:07:20.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:19 smithi023 ceph-mon[97824]: pgmap v557: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:07:20.313 INFO:tasks.check_counter:Daemon mds.b mds.exported=363 2023-11-17T00:07:20.314 INFO:tasks.check_counter:Daemon mds.b mds.imported=197 2023-11-17T00:07:20.314 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=1712 2023-11-17T00:07:20.314 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=568 2023-11-17T00:07:20.314 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=152 2023-11-17T00:07:20.314 DEBUG:tasks.check_counter:Getting stats from e 2023-11-17T00:07:20.314 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-11-17T00:07:20.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:19 smithi173 ceph-mon[131534]: pgmap v557: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-11-17T00:07:20.954 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-17T00:07:22.340 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 453, 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 455, 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 37869294, 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 140947, 2023-11-17T00:07:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.058166049, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.013023480, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.029123324, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.011769292, 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 453, 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.020161804, 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000044507 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 651, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 638, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 40654134, 2023-11-17T00:07:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 234094, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.084979177, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.018192217, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.045882271, 2023-11-17T00:07:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.018252556, 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 638, 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.032239427, 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000050532 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 502608, 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 85120 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 615, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 725, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 37948655, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 439402, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.085605617, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.021122395, 2023-11-17T00:07:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.035079597, 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.022237660, 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 723, 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.043154718, 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000059688 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 248448 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-11-17T00:07:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2538, 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.114509038, 2023-11-17T00:07:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000045117 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 18426, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 0, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 18426, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 18426, 2023-11-17T00:07:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 5321, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 1, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2023-11-17T00:07:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2023-11-17T00:07:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2023-11-17T00:07:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2023-11-17T00:07:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2023-11-17T00:07:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2023-11-17T00:07:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2023-11-17T00:07:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2023-11-17T00:07:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2023-11-17T00:07:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2023-11-17T00:07:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 0, 2023-11-17T00:07:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2023-11-17T00:07:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2023-11-17T00:07:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2023-11-17T00:07:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2023-11-17T00:07:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2023-11-17T00:07:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 67, 2023-11-17T00:07:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 0, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 119714805, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 50496 2023-11-17T00:07:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 13, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 18527, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 18514, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 11, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 1079, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 1068, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 18426, 2023-11-17T00:07:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 50381, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 31955, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 85652, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 207156 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.374 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.375 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.376 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.377 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.377 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2023-11-17T00:07:22.378 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.378 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.378 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.379 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.379 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:22.383 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.383 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.384 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.387 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.387 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2023-11-17T00:07:22.387 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.387 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.388 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.388 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.388 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:22.389 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.389 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.391 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2023-11-17T00:07:22.392 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.393 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.394 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.394 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.394 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.395 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.396 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.396 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2023-11-17T00:07:22.396 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.396 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.401 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.401 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.401 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2023-11-17T00:07:22.402 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.402 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.402 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.402 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.402 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.403 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.404 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.405 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.406 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.407 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.409 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.410 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.411 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 1, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2023-11-17T00:07:22.412 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 2370 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:22.413 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:22.414 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:22.415 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:22.416 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 221681, 2023-11-17T00:07:22.417 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 66, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:22.418 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 170, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:22.419 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-11-17T00:07:22.420 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 13014664, 2023-11-17T00:07:22.420 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 73832, 2023-11-17T00:07:22.420 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:22.420 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-11-17T00:07:22.420 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 1546, 2023-11-17T00:07:22.421 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 1546, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1546, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2.578007206, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001667533 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.422 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1546, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1547 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 1546, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 1546, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2023-11-17T00:07:22.423 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 931, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 613, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2023-11-17T00:07:22.424 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2023-11-17T00:07:22.425 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2023-11-17T00:07:22.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2023-11-17T00:07:22.427 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2023-11-17T00:07:22.428 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 5, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 9, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 9, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2023-11-17T00:07:22.429 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:22.430 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:22.431 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:22.432 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 1715, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 116306159, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 1715, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:22.433 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1715, 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 116306159, 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.434 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:22.435 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1546, 2023-11-17T00:07:22.436 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 115535385, 2023-11-17T00:07:22.436 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1546, 2023-11-17T00:07:22.436 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 115535385, 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.437 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:22.438 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:22.439 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:22.439 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:22.439 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1546, 2023-11-17T00:07:22.439 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1546, 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1546, 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1546, 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.440 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.441 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:22.441 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:22.446 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:22.446 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:22.447 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:22.447 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:22.447 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:22.447 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.448 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.449 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.452 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x5556d6946460": { 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:22.453 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:22.454 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:22.455 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:22.455 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:22.455 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.455 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:22.455 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-17T00:07:22.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:22 smithi156 ceph-mon[131105]: pgmap v558: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:22 smithi173 ceph-mon[131534]: pgmap v558: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:22.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:22 smithi023 ceph-mon[97824]: pgmap v558: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:22.817 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-11-17T00:07:22.817 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-11-17T00:07:22.817 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-11-17T00:07:22.817 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-11-17T00:07:22.817 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2023-11-17T00:07:22.818 DEBUG:tasks.check_counter:Getting stats from h 2023-11-17T00:07:22.818 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-11-17T00:07:23.464 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-17T00:07:24.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:24 smithi156 ceph-mon[131105]: pgmap v559: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:07:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:24 smithi173 ceph-mon[131534]: pgmap v559: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:07:24.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:24 smithi023 ceph-mon[97824]: pgmap v559: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-11-17T00:07:24.851 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-17T00:07:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 789, 2023-11-17T00:07:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 775, 2023-11-17T00:07:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 42656467, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 276589, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.103162957, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.028797924, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.046330606, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.022845557, 2023-11-17T00:07:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 774, 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.044404860, 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000057370 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 508928, 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 84080 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 252, 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 359, 2023-11-17T00:07:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 67194500, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 327151, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.052395650, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.008840459, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.034976074, 2023-11-17T00:07:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.005959503, 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 358, 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.023890156, 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000066732 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3152, 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 250272 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 859, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 861, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 93630565, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 267156, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2023-11-17T00:07:25.830 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.106143555, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.014322638, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.063746311, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.023226248, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 859, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.031676393, 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000036875 2023-11-17T00:07:25.831 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:25.832 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:25.833 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-11-17T00:07:25.833 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2023-11-17T00:07:25.833 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-11-17T00:07:25.833 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:25.833 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2023-11-17T00:07:26.586 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:26.586 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2783, 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.099445217, 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000035733 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:26.587 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2023-11-17T00:07:26.588 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2023-11-17T00:07:26.589 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 19297, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2023-11-17T00:07:26.590 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 0, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 19297, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 19297, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 4362, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2023-11-17T00:07:26.591 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2023-11-17T00:07:26.592 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:26.593 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:26.594 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2023-11-17T00:07:26.595 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2023-11-17T00:07:26.596 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2023-11-17T00:07:26.597 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:26.598 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:26.599 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 0, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2023-11-17T00:07:26.600 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 99, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2023-11-17T00:07:26.601 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 0, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 206644294, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.602 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 86673 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 20, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 30180, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 30160, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 19, 2023-11-17T00:07:26.603 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 1841, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 1822, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 19297, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 55629, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 36332, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2023-11-17T00:07:26.604 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 133672, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 207156 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2023-11-17T00:07:26.605 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.606 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.607 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2023-11-17T00:07:26.609 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.610 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.611 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.612 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2023-11-17T00:07:26.613 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.614 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.615 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.616 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.617 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:26.618 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.619 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:26.620 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.621 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.622 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.623 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.624 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:26.625 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2023-11-17T00:07:26.626 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 2372 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:26.627 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:26.628 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:26.629 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:26.630 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:26.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:26.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:26.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:26.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:26.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 239780, 2023-11-17T00:07:26.632 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 80, 2023-11-17T00:07:26.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:26.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:26.633 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-11-17T00:07:26.633 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-11-17T00:07:26.633 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:26.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:26.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:26.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:26.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:26.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 170, 2023-11-17T00:07:26.635 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:26.635 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:26.635 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:26.635 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-11-17T00:07:26.635 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 13653808, 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 77467, 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-11-17T00:07:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.637 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2023-11-17T00:07:26.637 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2023-11-17T00:07:26.637 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2023-11-17T00:07:26.637 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 1725, 2023-11-17T00:07:26.637 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 1725, 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1725, 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 5.307014701, 2023-11-17T00:07:26.638 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003076530 2023-11-17T00:07:26.639 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.639 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2023-11-17T00:07:26.639 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2023-11-17T00:07:26.639 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1725, 2023-11-17T00:07:26.639 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1726 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 1725, 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 1725, 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2023-11-17T00:07:26.640 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2023-11-17T00:07:27.114 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1053, 2023-11-17T00:07:27.174 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2023-11-17T00:07:27.174 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 670, 2023-11-17T00:07:27.174 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2023-11-17T00:07:27.175 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2023-11-17T00:07:27.175 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2023-11-17T00:07:27.232 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2023-11-17T00:07:27.233 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2023-11-17T00:07:27.233 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2023-11-17T00:07:27.233 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2023-11-17T00:07:27.234 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2023-11-17T00:07:27.234 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:27.234 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:27.234 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2023-11-17T00:07:27.235 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2023-11-17T00:07:27.366 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:27.367 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2023-11-17T00:07:27.368 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2023-11-17T00:07:27.369 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 5, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 11, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 11, 2023-11-17T00:07:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2023-11-17T00:07:27.371 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2023-11-17T00:07:27.371 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.426 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.427 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 1896, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 203299231, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 1896, 2023-11-17T00:07:27.428 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:27.429 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:27.429 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1896, 2023-11-17T00:07:27.429 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 203299231, 2023-11-17T00:07:27.429 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:27.429 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:27.605 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:27.606 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1725, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 202490360, 2023-11-17T00:07:27.607 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1725, 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 202490360, 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:27.608 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:27.609 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1725, 2023-11-17T00:07:27.813 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1725, 2023-11-17T00:07:27.813 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1725, 2023-11-17T00:07:27.813 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1725, 2023-11-17T00:07:27.814 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:27.814 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:27.814 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:27.814 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:28.153 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:28.154 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:28.155 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55dd05d48460": { 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-11-17T00:07:28.156 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-11-17T00:07:28.157 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:28.158 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-17T00:07:28.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:26 smithi023 ceph-mon[97824]: pgmap v560: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:28.176 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:26 smithi173 ceph-mon[131534]: pgmap v560: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:28.177 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2023-11-17T00:07:28.177 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2023-11-17T00:07:28.177 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2023-11-17T00:07:28.177 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2023-11-17T00:07:28.178 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2023-11-17T00:07:28.178 DEBUG:tasks.check_counter:Getting stats from k 2023-11-17T00:07:28.178 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-11-17T00:07:28.178 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:26 smithi156 ceph-mon[131105]: pgmap v560: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 8 op/s 2023-11-17T00:07:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:28 smithi173 ceph-mon[131534]: pgmap v561: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-17T00:07:28.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:28 smithi156 ceph-mon[131105]: pgmap v561: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-17T00:07:28.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:28 smithi023 ceph-mon[97824]: pgmap v561: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-17T00:07:28.832 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 4, 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 4, 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 2992, 2023-11-17T00:07:30.219 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 448, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.001585197, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000037847, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000701545, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:30.220 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000835871, 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000278623 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:30.221 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 448 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 155, 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 152, 2023-11-17T00:07:30.222 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 453728, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 85664, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.023557397, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.008764781, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.011339178, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.003965453, 2023-11-17T00:07:30.223 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 151, 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.007896412, 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000052294 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:30.224 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 453728, 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 85664 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2, 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 116, 2023-11-17T00:07:30.225 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 320, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 172144, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.006482489, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.004082915, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.001214099, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:30.226 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 113, 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.006611658, 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000058510 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:30.227 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 172144 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.229 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 0, 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:30.230 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:30.231 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:30.232 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:30.233 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:30.233 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:30.233 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:30.233 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:30.234 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:30.235 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 148967, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 29, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-11-17T00:07:30.237 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:30.238 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 2, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-11-17T00:07:30.239 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-17T00:07:30.240 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:30.240 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-11-17T00:07:30.240 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-11-17T00:07:30.240 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-11-17T00:07:30.240 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 157, 2023-11-17T00:07:30.242 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 431864, 2023-11-17T00:07:30.242 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:30.242 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 157, 2023-11-17T00:07:30.242 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-11-17T00:07:30.242 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 157, 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 431864, 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-11-17T00:07:30.243 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:30.244 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-17T00:07:30.244 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-17T00:07:30.244 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:29 smithi156 ceph-mon[131105]: pgmap v562: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-17T00:07:30.244 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:29 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:07:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:29 smithi023 ceph-mon[97824]: pgmap v562: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-17T00:07:30.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:29 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:07:30.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:29 smithi173 ceph-mon[131534]: pgmap v562: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-17T00:07:30.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:29 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T00:07:30.675 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-11-17T00:07:30.675 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-11-17T00:07:30.675 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2023-11-17T00:07:30.675 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2023-11-17T00:07:30.675 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2023-11-17T00:07:30.675 DEBUG:tasks.check_counter:Getting stats from c 2023-11-17T00:07:30.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-11-17T00:07:31.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.113 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.113 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.114 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Adjusting osd_memory_target on smithi023 to 221.9M 2023-11-17T00:07:31.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Unable to set osd_memory_target on smithi023 to 232772915: error parsing value: Value '232772915' is below minimum 939524096 2023-11-17T00:07:31.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:07:31.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:07:31.118 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.120 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:07:31.120 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Service mgr has no related daemon already placed. 2023-11-17T00:07:31.120 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.120 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:30 smithi173 ceph-mon[131534]: Service mds.all has no related daemon already placed. 2023-11-17T00:07:31.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Adjusting osd_memory_target on smithi023 to 221.9M 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Unable to set osd_memory_target on smithi023 to 232772915: error parsing value: Value '232772915' is below minimum 939524096 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Service mgr has no related daemon already placed. 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.122 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:30 smithi156 ceph-mon[131105]: Service mds.all has no related daemon already placed. 2023-11-17T00:07:31.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Adjusting osd_memory_target on smithi023 to 221.9M 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Unable to set osd_memory_target on smithi023 to 232772915: error parsing value: Value '232772915' is below minimum 939524096 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' 2023-11-17T00:07:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Service mgr has no related daemon already placed. 2023-11-17T00:07:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Service mon has no related daemon already placed. 2023-11-17T00:07:31.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:30 smithi023 ceph-mon[97824]: Service mds.all has no related daemon already placed. 2023-11-17T00:07:31.336 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-17T00:07:32.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:31 smithi023 ceph-mon[97824]: pgmap v563: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-17T00:07:32.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:31 smithi173 ceph-mon[131534]: pgmap v563: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-17T00:07:32.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:31 smithi156 ceph-mon[131105]: pgmap v563: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-11-17T00:07:32.591 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-17T00:07:32.592 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:32.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 0, 2023-11-17T00:07:32.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2, 2023-11-17T00:07:32.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 0, 2023-11-17T00:07:32.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 0, 2023-11-17T00:07:32.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2023-11-17T00:07:32.597 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2023-11-17T00:07:32.598 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.001452363, 2023-11-17T00:07:32.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000000000, 2023-11-17T00:07:32.600 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000350180, 2023-11-17T00:07:32.600 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:32.601 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:32.602 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:32.603 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:32.603 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:32.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.605 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:32.605 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:32.605 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:32.606 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:32.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.606 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:32.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:32.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.608 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:32.609 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 45, 2023-11-17T00:07:32.610 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 42, 2023-11-17T00:07:32.610 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 152400, 2023-11-17T00:07:32.611 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 21728, 2023-11-17T00:07:32.612 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 2, 2023-11-17T00:07:32.612 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:32.613 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.007293008, 2023-11-17T00:07:32.614 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.002216135, 2023-11-17T00:07:32.614 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.005007684, 2023-11-17T00:07:32.614 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000924494, 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 42, 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.004474933, 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000106546 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 152400, 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 21728 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1, 2023-11-17T00:07:32.616 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 29, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 160, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 80880, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 3, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.001727724, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000886930, 2023-11-17T00:07:32.617 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000498129, 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 27, 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001623922, 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000060145 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:32.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:32.619 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:32.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:32.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.619 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 80880 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 0, 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.620 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2023-11-17T00:07:32.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:32.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:32.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:32.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:32.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:32.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:32.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:32.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:32.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:32.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:32.626 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:32.626 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:32.626 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:32.627 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:32.628 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:32.628 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:32.628 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:32.628 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:32.629 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:32.629 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:32.629 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:32.629 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:32.630 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:32.630 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:32.630 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:32.630 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:32.630 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:32.631 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:32.631 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:32.633 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:32.633 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:32.633 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2023-11-17T00:07:32.633 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 149048, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 26, 2023-11-17T00:07:32.636 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:32.637 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 0, 2023-11-17T00:07:32.638 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 0, 2023-11-17T00:07:32.638 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:32.639 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:32.639 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:32.639 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 928, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2023-11-17T00:07:32.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 42, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 143654, 2023-11-17T00:07:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 42, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 42, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 143654, 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:32.656 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:32.656 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:32.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:32.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:32.656 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-17T00:07:32.991 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.c 2023-11-17T00:07:32.991 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.c 2023-11-17T00:07:32.992 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.c 2023-11-17T00:07:32.992 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.c 2023-11-17T00:07:32.992 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2023-11-17T00:07:32.992 DEBUG:tasks.check_counter:Getting stats from f 2023-11-17T00:07:32.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-11-17T00:07:33.636 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-17T00:07:34.425 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:34 smithi173 ceph-mon[131534]: pgmap v564: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:34.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:34 smithi156 ceph-mon[131105]: pgmap v564: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:34.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:34 smithi023 ceph-mon[97824]: pgmap v564: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:34.852 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 23386, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 30418, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 42153444, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 93981751, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2023-11-17T00:07:34.853 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.796678390, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.645225930, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.601591572, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.537794312, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 30416, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.153020565, 2023-11-17T00:07:34.854 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000103663 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 22194, 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003243051, 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000146 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:34.855 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 56285, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 112049, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 21552603, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 105452833, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 12, 2023-11-17T00:07:34.856 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 8, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 4.496783230, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 1.962028019, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 2.690885683, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.496101038, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 112049, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 11.763694981, 2023-11-17T00:07:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000104987 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 103935, 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.018488732, 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000177 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 17653120, 2023-11-17T00:07:34.858 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 58440768 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 20382, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 20727, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 45271701, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 78101866, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 8, 2023-11-17T00:07:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.407570570, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.427470069, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.516949463, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.479975042, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 20725, 2023-11-17T00:07:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.116754608, 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000150386 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 15163, 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002485822, 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000163 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 253456 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2023-11-17T00:07:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1352, 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 39.590536030, 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.029282940 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 31468, 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 10.353021552, 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000329001 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2023-11-17T00:07:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 36906, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 29513, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 29513, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 87.439643437, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002962750 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 7322, 2023-11-17T00:07:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 274, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 207, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 20, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 36, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 18, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 1, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 7, 2023-11-17T00:07:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 10, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 15, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 19699, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 2, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 2, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 5, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 47690, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 39679, 2023-11-17T00:07:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 7322, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 7, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 274, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 1, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 175, 2023-11-17T00:07:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 10642, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 348, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 246, 2023-11-17T00:07:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 1, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 345, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 20985, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 6940, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 6545, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 182, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 27523 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 161, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 3762, 2023-11-17T00:07:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 1883, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 1888, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 161, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 18416, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 18416, 2023-11-17T00:07:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 188, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 56, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 50496, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2023-11-17T00:07:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 50496, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 67, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2023-11-17T00:07:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 67, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 119715943, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 871, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 91.975705931, 2023-11-17T00:07:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.105597825 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 23, 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 21936, 2023-11-17T00:07:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 21913, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 21, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 2771, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 2750, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 18, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 53012, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 52994, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 2, 2023-11-17T00:07:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 25617, 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 25615, 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 129984, 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 198964 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 39203, 2023-11-17T00:07:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 340, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 36906, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 5, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 793, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2114, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.811316991, 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001329856 2023-11-17T00:07:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 28, 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.035135190, 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001254828 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8259, 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 21.878624367, 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002649064 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000320209, 2023-11-17T00:07:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000320209 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2023-11-17T00:07:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 695, 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 7.002282034, 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.010075225 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2023-11-17T00:07:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2023-11-17T00:07:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.049869203, 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.024934601 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2023-11-17T00:07:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2023-11-17T00:07:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18414, 2023-11-17T00:07:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 55.662095443, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003022813 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2023-11-17T00:07:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 12190, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 12190, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 376 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 301851, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 148, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 1760, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 20, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2023-11-17T00:07:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 170, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 92239, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 517, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 39804, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 119654089, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 33, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 39804, 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2023-11-17T00:07:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 39804, 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 384.578153864, 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.009661796 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 39804, 2023-11-17T00:07:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 40329 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 39804, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 552, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 39252, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2023-11-17T00:07:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 278, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 1150, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 96, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2023-11-17T00:07:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 37750, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2023-11-17T00:07:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2023-11-17T00:07:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 93, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 5, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2023-11-17T00:07:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 443, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 548, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 34 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 37426 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 16, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 8, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 18416, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 100049, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 97933230, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 100049, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 100049, 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 97933230, 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 39804, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 121657097, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 39804, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 121657097, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 39804, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 39804, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 39804, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 39804, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 18416, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 2007392, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 18416, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 278, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2007392, 2023-11-17T00:07:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55a222dce0a0": { 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 50496, 2023-11-17T00:07:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 115521639, 2023-11-17T00:07:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 50496, 2023-11-17T00:07:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 871, 2023-11-17T00:07:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 115521639, 2023-11-17T00:07:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:34.934 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-17T00:07:35.290 INFO:tasks.check_counter:Daemon mds.f mds.exported=175 2023-11-17T00:07:35.290 INFO:tasks.check_counter:Daemon mds.f mds.imported=348 2023-11-17T00:07:35.290 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=161 2023-11-17T00:07:35.291 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=3762 2023-11-17T00:07:35.291 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=20 2023-11-17T00:07:35.291 DEBUG:tasks.check_counter:Getting stats from i 2023-11-17T00:07:35.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-11-17T00:07:35.931 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-17T00:07:36.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:36 smithi173 ceph-mon[131534]: pgmap v565: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 8 op/s 2023-11-17T00:07:36.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:36 smithi156 ceph-mon[131105]: pgmap v565: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 8 op/s 2023-11-17T00:07:36.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:36 smithi023 ceph-mon[97824]: pgmap v565: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 8 op/s 2023-11-17T00:07:37.289 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-17T00:07:37.289 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 27417, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 20374, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 56782064, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 83741002, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2023-11-17T00:07:37.290 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.591303919, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.625071011, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.638799967, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.398792138, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 20373, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.436596873, 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000119599 2023-11-17T00:07:37.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.292 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:37.292 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 22678, 2023-11-17T00:07:37.292 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002641948, 2023-11-17T00:07:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000116 2023-11-17T00:07:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 518416, 2023-11-17T00:07:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 92624 2023-11-17T00:07:37.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 20731, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 21419, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 46224651, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 102848544, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 8, 2023-11-17T00:07:37.294 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.614876444, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.540932563, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.533795191, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.559615517, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 21419, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.286215720, 2023-11-17T00:07:37.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000153425 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 16961, 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002783127, 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000164 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:37.296 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 253280 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 111623, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 211286, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 44955948, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 212873076, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 13, 2023-11-17T00:07:37.297 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 10, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 7.747179898, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 3.499696782, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 4.484833755, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.722903808, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 211283, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 18.826611593, 2023-11-17T00:07:37.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000089106 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 226677, 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.036548901, 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000161 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 36282208, 2023-11-17T00:07:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 124142688 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2442, 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 78.991570750, 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.032347080 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:37.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 33557, 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 11.308639821, 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000336997 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 66101, 2023-11-17T00:07:37.302 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 60623, 2023-11-17T00:07:37.303 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 60623, 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 224.086747169, 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003696398 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2023-11-17T00:07:37.304 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 5478, 2023-11-17T00:07:37.305 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 476, 2023-11-17T00:07:37.312 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:37.312 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 1619, 2023-11-17T00:07:37.312 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 193, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 64, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 19, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 1, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 8, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 10, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 17, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 29953, 2023-11-17T00:07:37.313 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 3, 2023-11-17T00:07:37.314 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 3, 2023-11-17T00:07:37.314 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 5, 2023-11-17T00:07:37.314 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 87071, 2023-11-17T00:07:37.314 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 81520, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 4931, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 5, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 181, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 8, 2023-11-17T00:07:37.315 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 71, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 24417, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 64, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2023-11-17T00:07:37.316 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 246, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:37.317 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 4, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 107, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 40725, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 25574, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 25208, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 608, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 34654 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 1306, 2023-11-17T00:07:37.319 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 2036, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 1020, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 1182, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 1610, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 19280, 2023-11-17T00:07:37.320 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 19280, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2023-11-17T00:07:37.321 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 232, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 266, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.322 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 86673, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 86673, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2023-11-17T00:07:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 99, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 99, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 206644893, 2023-11-17T00:07:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1273, 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 137.516498918, 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.108025529 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.325 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 24, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 31123, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 31099, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 23, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 2953, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 2930, 2023-11-17T00:07:37.326 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 19, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 52542, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 52523, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 3, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 47054, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 47051, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 180440, 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 198964 2023-11-17T00:07:37.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 69028, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 11, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 66101, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 18, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 371, 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2023-11-17T00:07:37.328 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 20945, 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 83.424080500, 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003983006 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 368, 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.498304706, 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001354088 2023-11-17T00:07:37.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.330 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18977, 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 43.554763250, 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002295134 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.331 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:37.333 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.334 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002201343, 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001100671 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.336 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.337 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1030, 2023-11-17T00:07:37.338 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 14.398881386, 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.013979496 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:37.340 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.067004601, 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.033502300 2023-11-17T00:07:37.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3, 2023-11-17T00:07:37.342 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002353163, 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000784387 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.343 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10, 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 8.258260319, 2023-11-17T00:07:37.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.825826031 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2023-11-17T00:07:37.345 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8, 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.008126680, 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001015835 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 19278, 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 73.872771221, 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003831972 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:37.347 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.349 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2023-11-17T00:07:37.349 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2023-11-17T00:07:37.349 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 19118, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 19118, 2023-11-17T00:07:37.350 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 329 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:37.351 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:37.352 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:37.353 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:37.354 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:37.355 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 283437, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 118, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 264, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 3, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:37.356 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 170, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:37.357 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 99072, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 605, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 43915, 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 220837238, 2023-11-17T00:07:37.359 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 33, 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 43915, 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 43915, 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 418.289379661, 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.009524977 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.360 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 43915, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 46031 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 43915, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 754, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 43161, 2023-11-17T00:07:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 278, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 1551, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 115, 2023-11-17T00:07:37.362 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 39824, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:37.363 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2023-11-17T00:07:37.364 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2023-11-17T00:07:37.365 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2023-11-17T00:07:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 93, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2023-11-17T00:07:37.367 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 5, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 3248, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 952, 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 33 2023-11-17T00:07:37.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2023-11-17T00:07:37.369 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 66610 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 16, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 8, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 19280, 2023-11-17T00:07:37.370 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 159767, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 129192569, 2023-11-17T00:07:37.371 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 159767, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 159767, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 129192569, 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:37.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:37.373 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 43915, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 222931158, 2023-11-17T00:07:37.374 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 43915, 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 222931158, 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.375 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:37.376 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 43915, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 43915, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 43915, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 43915, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 19280, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 2101536, 2023-11-17T00:07:37.378 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 19280, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 278, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2101536, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.379 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55b144c7a0a0": { 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:37.380 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 86673, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 202450589, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 86673, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1273, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 202450589, 2023-11-17T00:07:37.381 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:37.382 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-17T00:07:37.671 INFO:tasks.check_counter:Daemon mds.i mds.exported=71 2023-11-17T00:07:37.672 INFO:tasks.check_counter:Daemon mds.i mds.imported=64 2023-11-17T00:07:37.672 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=1306 2023-11-17T00:07:37.672 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=2036 2023-11-17T00:07:37.672 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=193 2023-11-17T00:07:37.672 DEBUG:tasks.check_counter:Getting stats from l 2023-11-17T00:07:37.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1c9565146c8647d4e00eb6c7ae626dc192680e7b shell --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-11-17T00:07:38.313 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-17T00:07:38.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:38 smithi173 ceph-mon[131534]: pgmap v566: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:38.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:38 smithi156 ceph-mon[131105]: pgmap v566: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:38.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:38 smithi023 ceph-mon[97824]: pgmap v566: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-11-17T00:07:39.597 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-17T00:07:39.598 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2023-11-17T00:07:39.598 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 153, 2023-11-17T00:07:39.598 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 150, 2023-11-17T00:07:39.598 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 201264, 2023-11-17T00:07:39.598 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 53041, 2023-11-17T00:07:39.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2023-11-17T00:07:39.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2023-11-17T00:07:39.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.020370713, 2023-11-17T00:07:39.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005146623, 2023-11-17T00:07:39.599 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.008365932, 2023-11-17T00:07:39.600 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.005558374, 2023-11-17T00:07:39.600 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:39.600 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 150, 2023-11-17T00:07:39.600 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.019720310, 2023-11-17T00:07:39.600 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000131468 2023-11-17T00:07:39.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.601 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:39.601 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.601 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.602 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.602 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 164816, 2023-11-17T00:07:39.602 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 19760 2023-11-17T00:07:39.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2023-11-17T00:07:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 206, 2023-11-17T00:07:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 230, 2023-11-17T00:07:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 8354723, 2023-11-17T00:07:39.603 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 185380, 2023-11-17T00:07:39.604 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2023-11-17T00:07:39.604 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2023-11-17T00:07:39.604 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.021086954, 2023-11-17T00:07:39.604 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005712937, 2023-11-17T00:07:39.604 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.008468234, 2023-11-17T00:07:39.605 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.005334667, 2023-11-17T00:07:39.605 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:39.605 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 228, 2023-11-17T00:07:39.605 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.016170977, 2023-11-17T00:07:39.605 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000070925 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 121568 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 63, 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 65, 2023-11-17T00:07:39.607 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 8405358, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 19609, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.009991896, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.001190960, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.006687867, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.001631792, 2023-11-17T00:07:39.608 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 63, 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006435994, 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000102158 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2023-11-17T00:07:39.610 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 625, 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.110369523, 2023-11-17T00:07:39.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000176591 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2023-11-17T00:07:39.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2023-11-17T00:07:39.614 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 7424, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 5189, 2023-11-17T00:07:39.615 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 2229, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 6, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 10, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 7770, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 3, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2023-11-17T00:07:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2023-11-17T00:07:39.617 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2023-11-17T00:07:39.617 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2023-11-17T00:07:39.617 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2023-11-17T00:07:39.617 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2023-11-17T00:07:39.617 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2023-11-17T00:07:39.618 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 246, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2023-11-17T00:07:39.619 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2023-11-17T00:07:39.620 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-17T00:07:39.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.622 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2023-11-17T00:07:39.623 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2023-11-17T00:07:39.624 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2023-11-17T00:07:39.625 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2023-11-17T00:07:39.626 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2023-11-17T00:07:39.627 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 182091496, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 0, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 182114968, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 7248 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 20, 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3427, 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 3407, 2023-11-17T00:07:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 31, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 356, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 325, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 7424, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 17908, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 10484, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2023-11-17T00:07:39.630 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 57736, 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 207156 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2023-11-17T00:07:39.631 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2023-11-17T00:07:39.632 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2023-11-17T00:07:39.632 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2023-11-17T00:07:39.632 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2023-11-17T00:07:39.632 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.633 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.633 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.633 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.634 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.635 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2023-11-17T00:07:39.637 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.638 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.641 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.642 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2023-11-17T00:07:39.642 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.643 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.643 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2023-11-17T00:07:39.644 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.645 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.645 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.646 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2023-11-17T00:07:39.648 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.649 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2023-11-17T00:07:39.651 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.652 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.653 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.654 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2023-11-17T00:07:39.656 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2023-11-17T00:07:39.657 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.658 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2023-11-17T00:07:39.659 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2023-11-17T00:07:39.659 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2023-11-17T00:07:39.659 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2023-11-17T00:07:39.659 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 0, 2023-11-17T00:07:39.659 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2023-11-17T00:07:39.660 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2402 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2023-11-17T00:07:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-17T00:07:39.662 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2023-11-17T00:07:39.663 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2023-11-17T00:07:39.663 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2023-11-17T00:07:39.663 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-17T00:07:39.663 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2023-11-17T00:07:39.663 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2023-11-17T00:07:39.664 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2023-11-17T00:07:39.664 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2023-11-17T00:07:39.664 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2023-11-17T00:07:39.665 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-17T00:07:39.665 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2023-11-17T00:07:39.665 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-17T00:07:39.667 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2023-11-17T00:07:39.668 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2023-11-17T00:07:39.669 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2023-11-17T00:07:39.670 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2023-11-17T00:07:39.670 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2023-11-17T00:07:39.670 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2023-11-17T00:07:39.670 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2023-11-17T00:07:39.671 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2023-11-17T00:07:39.671 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2023-11-17T00:07:39.671 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 196589, 2023-11-17T00:07:39.671 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 51, 2023-11-17T00:07:39.671 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2023-11-17T00:07:39.672 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8888, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 170, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 5198219, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 15950, 2023-11-17T00:07:39.673 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2023-11-17T00:07:39.674 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 375, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 26, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 4, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 375, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 375, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.297993148, 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000794648 2023-11-17T00:07:39.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 375, 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 376 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.676 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 375, 2023-11-17T00:07:39.677 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 375, 2023-11-17T00:07:39.677 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2023-11-17T00:07:39.677 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2023-11-17T00:07:39.677 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 212, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 160, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2023-11-17T00:07:39.678 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2023-11-17T00:07:39.679 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2023-11-17T00:07:39.679 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2023-11-17T00:07:39.679 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2023-11-17T00:07:39.679 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2023-11-17T00:07:39.679 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2023-11-17T00:07:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 2, 2023-11-17T00:07:39.681 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2023-11-17T00:07:39.682 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2023-11-17T00:07:39.682 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2023-11-17T00:07:39.683 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2023-11-17T00:07:39.683 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2023-11-17T00:07:39.683 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2023-11-17T00:07:39.684 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2023-11-17T00:07:39.684 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2023-11-17T00:07:39.684 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2023-11-17T00:07:39.685 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2023-11-17T00:07:39.686 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 0, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 6, 2023-11-17T00:07:39.687 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 6, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2023-11-17T00:07:39.688 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2023-11-17T00:07:39.689 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-17T00:07:39.690 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 418, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 16918582, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 418, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:39.691 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 418, 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 16918582, 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.693 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2023-11-17T00:07:39.693 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:39.694 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 375, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 16685963, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 375, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 16685963, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:39.696 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 375, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 375, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 375, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 375, 2023-11-17T00:07:39.697 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.698 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:39.699 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:39.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.701 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.701 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55b1e9a24460": { 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2023-11-17T00:07:39.702 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2023-11-17T00:07:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2023-11-17T00:07:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2023-11-17T00:07:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2023-11-17T00:07:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-17T00:07:39.704 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-17T00:07:39.968 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2023-11-17T00:07:39.968 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2023-11-17T00:07:39.972 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2023-11-17T00:07:39.973 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2023-11-17T00:07:39.973 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2023-11-17T00:07:39.976 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-17T00:07:39.997 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-17T00:07:39.998 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-17T00:07:39.998 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-17T00:07:40.065 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-11-17T00:07:40.065 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-17T00:07:40.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:39 smithi173 ceph-mon[131534]: pgmap v567: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2023-11-17T00:07:40.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:07:40.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:39 smithi023 ceph-mon[97824]: pgmap v567: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2023-11-17T00:07:40.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:39 smithi156 ceph-mon[131105]: pgmap v567: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2023-11-17T00:07:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 -- 172.21.15.23:0/127535166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c101e10 msgr2=0x7fd60c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 --2- 172.21.15.23:0/127535166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c101e10 0x7fd60c1086d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd60400b0a0 tx=0x7fd60400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 -- 172.21.15.23:0/127535166 shutdown_connections 2023-11-17T00:07:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 --2- 172.21.15.23:0/127535166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c101e10 0x7fd60c1086d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 --2- 172.21.15.23:0/127535166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c100290 0x7fd60c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 --2- 172.21.15.23:0/127535166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101480 0x7fd60c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 -- 172.21.15.23:0/127535166 >> 172.21.15.23:0/127535166 conn(0x7fd60c0fb870 msgr2=0x7fd60c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 -- 172.21.15.23:0/127535166 shutdown_connections 2023-11-17T00:07:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.402+0000 7fd613dff700 1 -- 172.21.15.23:0/127535166 wait complete. 2023-11-17T00:07:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.403+0000 7fd613dff700 1 Processor -- start 2023-11-17T00:07:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 -- start start 2023-11-17T00:07:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c101480 0x7fd60c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101e10 0x7fd60c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c073bc0 con 0x7fd60c100290 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c073d30 con 0x7fd60c101e10 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd613dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c1a00f0 con 0x7fd60c101480 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd611b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd611b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47022/0 (socket says 172.21.15.23:47022) 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd611b9b700 1 -- 172.21.15.23:0/701670465 learned_addr learned my addr 172.21.15.23:0/701670465 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd61139a700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c101480 0x7fd60c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.404+0000 7fd61239c700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101e10 0x7fd60c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 -- 172.21.15.23:0/701670465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c101480 msgr2=0x7fd60c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c101480 0x7fd60c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 -- 172.21.15.23:0/701670465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101e10 msgr2=0x7fd60c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101e10 0x7fd60c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 -- 172.21.15.23:0/701670465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604009d00 con 0x7fd60c100290 2023-11-17T00:07:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd611b9b700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e970 tx=0x7fd60800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608009790 con 0x7fd60c100290 2023-11-17T00:07:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd60800c7e0 con 0x7fd60c100290 2023-11-17T00:07:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608003c70 con 0x7fd60c100290 2023-11-17T00:07:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c10bc70 con 0x7fd60c100290 2023-11-17T00:07:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.405+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c10bae0 con 0x7fd60c100290 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.407+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c1a0810 con 0x7fd60c100290 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.409+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd60800cc50 con 0x7fd60c100290 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.410+0000 7fd602ffd700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 0x7fd5f80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.410+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fd608014070 con 0x7fd60c100290 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.410+0000 7fd61139a700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 0x7fd5f80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.411+0000 7fd61139a700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 0x7fd5f80a3680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009c60 tx=0x7fd5fc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:40.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.414+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd608096170 con 0x7fd60c100290 2023-11-17T00:07:40.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.623+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd60c107100 con 0x7fd60c100290 2023-11-17T00:07:40.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.625+0000 7fd602ffd700 1 -- 172.21.15.23:0/701670465 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24603 (secure 0 0 0) 0x7fd608003dd0 con 0x7fd60c100290 2023-11-17T00:07:40.627 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:07:40.627 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:05:51.321983+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.627+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 msgr2=0x7fd5f80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.627+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 0x7fd5f80a3680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009c60 tx=0x7fd5fc009260 comp rx=0 tx=0).stop 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.627+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 msgr2=0x7fd60c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.627+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e970 tx=0x7fd60800ec80 comp rx=0 tx=0).stop 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 shutdown_connections 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c101e10 0x7fd60c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd60c100290 0x7fd60c0748a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fd5f80a11d0 0x7fd5f80a3680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 --2- 172.21.15.23:0/701670465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd60c101480 0x7fd60c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 >> 172.21.15.23:0/701670465 conn(0x7fd60c0fb870 msgr2=0x7fd60c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 shutdown_connections 2023-11-17T00:07:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.628+0000 7fd613dff700 1 -- 172.21.15.23:0/701670465 wait complete. 2023-11-17T00:07:40.649 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T23:54:03.596917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T23:58:44.557031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T23:58:45.676734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T23:58:49.239193+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-17T00:07:40.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.857+0000 7ff2769ff700 1 -- 172.21.15.23:0/1985075096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 msgr2=0x7ff2701031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.857+0000 7ff2769ff700 1 --2- 172.21.15.23:0/1985075096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701031c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff260009990 tx=0x7ff260009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 -- 172.21.15.23:0/1985075096 shutdown_connections 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 --2- 172.21.15.23:0/1985075096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270106020 0x7ff270108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 --2- 172.21.15.23:0/1985075096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701031c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 --2- 172.21.15.23:0/1985075096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270103700 0x7ff270105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 -- 172.21.15.23:0/1985075096 >> 172.21.15.23:0/1985075096 conn(0x7ff2700fa7c0 msgr2=0x7ff2700fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 -- 172.21.15.23:0/1985075096 shutdown_connections 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.858+0000 7ff2769ff700 1 -- 172.21.15.23:0/1985075096 wait complete. 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 Processor -- start 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 -- start start 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270103700 0x7ff270198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270106020 0x7ff27019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27019fe60 con 0x7ff270100de0 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27019ffa0 con 0x7ff270103700 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.859+0000 7ff2769ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2701a0130 con 0x7ff270106020 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47048/0 (socket says 172.21.15.23:47048) 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 -- 172.21.15.23:0/2539953406 learned_addr learned my addr 172.21.15.23:0/2539953406 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26f7fe700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270103700 0x7ff270198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff274f9c700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270106020 0x7ff27019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 -- 172.21.15.23:0/2539953406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270106020 msgr2=0x7ff27019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270106020 0x7ff27019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 -- 172.21.15.23:0/2539953406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270103700 msgr2=0x7ff270198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270103700 0x7ff270198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 -- 172.21.15.23:0/2539953406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2600095f0 con 0x7ff270100de0 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26ffff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff2600092e0 tx=0x7ff260009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26001d070 con 0x7ff270100de0 2023-11-17T00:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.860+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2600047e0 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.861+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2600053d0 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.861+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27010bc70 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.861+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27010bae0 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.862+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff270107b90 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.864+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff260004950 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.865+0000 7ff26d7fa700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 0x7ff2580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.865+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7ff2600cdc50 con 0x7ff270100de0 2023-11-17T00:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.865+0000 7ff26f7fe700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 0x7ff2580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:40.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.866+0000 7ff26f7fe700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 0x7ff2580a3680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff264009c60 tx=0x7ff264009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:40.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:40.869+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff260097080 con 0x7ff270100de0 2023-11-17T00:07:41.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.079+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff270061c40 con 0x7ff270100de0 2023-11-17T00:07:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.080+0000 7ff26d7fa700 1 -- 172.21.15.23:0/2539953406 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff26009a6a0 con 0x7ff270100de0 2023-11-17T00:07:41.081 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-11-17T00:07:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 msgr2=0x7ff2580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 0x7ff2580a3680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff264009c60 tx=0x7ff264009260 comp rx=0 tx=0).stop 2023-11-17T00:07:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 msgr2=0x7ff2701982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff2600092e0 tx=0x7ff260009580 comp rx=0 tx=0).stop 2023-11-17T00:07:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 shutdown_connections 2023-11-17T00:07:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7ff2580a11d0 0x7ff2580a3680 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff270106020 0x7ff27019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff270100de0 0x7ff2701982e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 --2- 172.21.15.23:0/2539953406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff270103700 0x7ff270198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 >> 172.21.15.23:0/2539953406 conn(0x7ff2700fa7c0 msgr2=0x7ff2700ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 shutdown_connections 2023-11-17T00:07:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.082+0000 7ff2769ff700 1 -- 172.21.15.23:0/2539953406 wait complete. 2023-11-17T00:07:41.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-17T00:07:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:40 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/701670465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 -- 172.21.15.23:0/3769199949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1002c0 msgr2=0x7fed4c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 --2- 172.21.15.23:0/3769199949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1002c0 0x7fed4c1006d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009990 tx=0x7fed3c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 -- 172.21.15.23:0/3769199949 shutdown_connections 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 --2- 172.21.15.23:0/3769199949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 --2- 172.21.15.23:0/3769199949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1002c0 0x7fed4c1006d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 --2- 172.21.15.23:0/3769199949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1014b0 0x7fed4c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 -- 172.21.15.23:0/3769199949 >> 172.21.15.23:0/3769199949 conn(0x7fed4c0fb8a0 msgr2=0x7fed4c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 -- 172.21.15.23:0/3769199949 shutdown_connections 2023-11-17T00:07:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.315+0000 7fed51dff700 1 -- 172.21.15.23:0/3769199949 wait complete. 2023-11-17T00:07:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.316+0000 7fed51dff700 1 Processor -- start 2023-11-17T00:07:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.316+0000 7fed51dff700 1 -- start start 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 0x7fed4c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c19fe00 con 0x7fed4c1014b0 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c19ff40 con 0x7fed4c1002c0 2023-11-17T00:07:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed51dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c1a00d0 con 0x7fed4c101e40 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47934/0 (socket says 172.21.15.23:47934) 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 -- 172.21.15.23:0/1224042637 learned_addr learned my addr 172.21.15.23:0/1224042637 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed4b7fe700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 0x7fed4c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed4bfff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 -- 172.21.15.23:0/1224042637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 msgr2=0x7fed4c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 -- 172.21.15.23:0/1224042637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 msgr2=0x7fed4c198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 0x7fed4c198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed43fff700 1 -- 172.21.15.23:0/1224042637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c0095f0 con 0x7fed4c1014b0 2023-11-17T00:07:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed4bfff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.317+0000 7fed4b7fe700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 0x7fed4c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed43fff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed3400d980 tx=0x7fed3400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3400bb60 con 0x7fed4c1014b0 2023-11-17T00:07:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c10bc90 con 0x7fed4c1014b0 2023-11-17T00:07:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed34007640 con 0x7fed4c1014b0 2023-11-17T00:07:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed34004e90 con 0x7fed4c1014b0 2023-11-17T00:07:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.318+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c10bb00 con 0x7fed4c1014b0 2023-11-17T00:07:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.319+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c1929f0 con 0x7fed4c1014b0 2023-11-17T00:07:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.321+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fed3400bcc0 con 0x7fed4c1014b0 2023-11-17T00:07:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.322+0000 7fed497fa700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 0x7fed380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.323+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fed340ccbc0 con 0x7fed4c1014b0 2023-11-17T00:07:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.323+0000 7fed4b7fe700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 0x7fed380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.323+0000 7fed4b7fe700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 0x7fed380a3600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.326+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed340999f0 con 0x7fed4c1014b0 2023-11-17T00:07:41.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:40 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/701670465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:41.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:40 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/701670465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:41.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.535+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed4c061cc0 con 0x7fed4c1014b0 2023-11-17T00:07:41.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.536+0000 7fed497fa700 1 -- 172.21.15.23:0/1224042637 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed340160a0 con 0x7fed4c1014b0 2023-11-17T00:07:41.537 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:07:41.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 msgr2=0x7fed380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 0x7fed380a3600 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c005fb0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 msgr2=0x7fed4c1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed3400d980 tx=0x7fed3400dd40 comp rx=0 tx=0).stop 2023-11-17T00:07:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 shutdown_connections 2023-11-17T00:07:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fed380a1150 0x7fed380a3600 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed4c101e40 0x7fed4c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed4c1002c0 0x7fed4c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 --2- 172.21.15.23:0/1224042637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed4c1014b0 0x7fed4c1987c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 >> 172.21.15.23:0/1224042637 conn(0x7fed4c0fb8a0 msgr2=0x7fed4c106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 shutdown_connections 2023-11-17T00:07:41.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.538+0000 7fed51dff700 1 -- 172.21.15.23:0/1224042637 wait complete. 2023-11-17T00:07:41.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-17T00:07:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 -- 172.21.15.23:0/3966588126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d24101e40 msgr2=0x7f8d24108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3966588126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d24101e40 0x7f8d24108700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00b0a0 tx=0x7f8d1c00b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 -- 172.21.15.23:0/3966588126 shutdown_connections 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3966588126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d24101e40 0x7f8d24108700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3966588126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d241002c0 0x7f8d241006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3966588126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241014b0 0x7f8d24101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 -- 172.21.15.23:0/3966588126 >> 172.21.15.23:0/3966588126 conn(0x7f8d240fb8a0 msgr2=0x7f8d240fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 -- 172.21.15.23:0/3966588126 shutdown_connections 2023-11-17T00:07:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.769+0000 7f8d28dff700 1 -- 172.21.15.23:0/3966588126 wait complete. 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.770+0000 7f8d28dff700 1 Processor -- start 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.770+0000 7f8d28dff700 1 -- start start 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 0x7f8d240748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d24101e40 0x7f8d24073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24073bc0 con 0x7f8d241014b0 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24073d30 con 0x7f8d24101e40 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d28dff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d241a0230 con 0x7f8d241002c0 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d21d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 0x7f8d240748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d21d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47952/0 (socket says 172.21.15.23:47952) 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d21d9b700 1 -- 172.21.15.23:0/3638711517 learned_addr learned my addr 172.21.15.23:0/3638711517 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d22d9d700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d24101e40 0x7f8d24073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 -- 172.21.15.23:0/3638711517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d24101e40 msgr2=0x7f8d24073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d24101e40 0x7f8d24073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 -- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 msgr2=0x7f8d24072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d2259c700 1 -- 172.21.15.23:0/3638711517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c009d00 con 0x7f8d241002c0 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.771+0000 7f8d21d9b700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d2259c700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 0x7f8d240748a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400e9c0 tx=0x7f8d1400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14004220 con 0x7f8d241002c0 2023-11-17T00:07:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2410bc90 con 0x7f8d241002c0 2023-11-17T00:07:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d14004380 con 0x7f8d241002c0 2023-11-17T00:07:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14003be0 con 0x7f8d241002c0 2023-11-17T00:07:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.772+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2410bb00 con 0x7f8d241002c0 2023-11-17T00:07:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.773+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d241077b0 con 0x7f8d241002c0 2023-11-17T00:07:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.775+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8d14003d40 con 0x7f8d241002c0 2023-11-17T00:07:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.776+0000 7f8d137fe700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 0x7f8d0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.777+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f8d14014070 con 0x7f8d241002c0 2023-11-17T00:07:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.777+0000 7f8d21d9b700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 0x7f8d0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.777+0000 7f8d21d9b700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 0x7f8d0c0a3600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d18011550 tx=0x7f8d18009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:41.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.779+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8d14096490 con 0x7f8d241002c0 2023-11-17T00:07:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.990+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d24061700 con 0x7f8d241002c0 2023-11-17T00:07:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.991+0000 7f8d137fe700 1 -- 172.21.15.23:0/3638711517 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d1401f340 con 0x7f8d241002c0 2023-11-17T00:07:41.992 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-11-17T00:07:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 msgr2=0x7f8d0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 0x7f8d0c0a3600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d18011550 tx=0x7f8d18009260 comp rx=0 tx=0).stop 2023-11-17T00:07:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 msgr2=0x7f8d240748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 0x7f8d240748a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400e9c0 tx=0x7f8d1400ecd0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 shutdown_connections 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d24101e40 0x7f8d24073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f8d0c0a1150 0x7f8d0c0a3600 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d241002c0 0x7f8d240748a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 --2- 172.21.15.23:0/3638711517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d241014b0 0x7f8d24072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 >> 172.21.15.23:0/3638711517 conn(0x7f8d240fb8a0 msgr2=0x7f8d241048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 shutdown_connections 2023-11-17T00:07:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:41.994+0000 7f8d28dff700 1 -- 172.21.15.23:0/3638711517 wait complete. 2023-11-17T00:07:42.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-17T00:07:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 -- 172.21.15.23:0/3181242930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04074b30 msgr2=0x7fcf04072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 --2- 172.21.15.23:0/3181242930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04074b30 0x7fcf04072f80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009990 tx=0x7fcef4009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 -- 172.21.15.23:0/3181242930 shutdown_connections 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 --2- 172.21.15.23:0/3181242930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04108190 0x7fcf0410a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 --2- 172.21.15.23:0/3181242930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04074b30 0x7fcf04072f80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 --2- 172.21.15.23:0/3181242930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04073550 0x7fcf040739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 -- 172.21.15.23:0/3181242930 >> 172.21.15.23:0/3181242930 conn(0x7fcf040fb8b0 msgr2=0x7fcf040fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 -- 172.21.15.23:0/3181242930 shutdown_connections 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.221+0000 7fcf09bff700 1 -- 172.21.15.23:0/3181242930 wait complete. 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.222+0000 7fcf09bff700 1 Processor -- start 2023-11-17T00:07:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 -- start start 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04074b30 0x7fcf041987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04108190 0x7fcf0419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419fe20 con 0x7fcf04073550 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419ff60 con 0x7fcf04074b30 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf09bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf041a00f0 con 0x7fcf04108190 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47976/0 (socket says 172.21.15.23:47976) 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf037fe700 1 -- 172.21.15.23:0/1748232693 learned_addr learned my addr 172.21.15.23:0/1748232693 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf02ffd700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04074b30 0x7fcf041987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.223+0000 7fcf03fff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04108190 0x7fcf0419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 -- 172.21.15.23:0/1748232693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04108190 msgr2=0x7fcf0419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04108190 0x7fcf0419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 -- 172.21.15.23:0/1748232693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04074b30 msgr2=0x7fcf041987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04074b30 0x7fcf041987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 -- 172.21.15.23:0/1748232693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef40095f0 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf037fe700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b680 tx=0x7fcef4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef401d070 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcef4021410 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0410de90 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4017650 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.224+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0410dd00 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.225+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf04106840 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.227+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcef4021580 con 0x7fcf04073550 2023-11-17T00:07:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.229+0000 7fcf00ff9700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.229+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fcef40ce820 con 0x7fcf04073550 2023-11-17T00:07:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.229+0000 7fcf02ffd700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.230+0000 7fcf02ffd700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 0x7fceec0a3600 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcf04074850 tx=0x7fcef800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.230+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcef4097c50 con 0x7fcf04073550 2023-11-17T00:07:42.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:41 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2539953406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:41 smithi023 ceph-mon[97824]: pgmap v568: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-17T00:07:42.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:41 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1224042637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:41 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2539953406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:41 smithi173 ceph-mon[131534]: pgmap v568: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-17T00:07:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:41 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1224042637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:41 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2539953406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:41 smithi156 ceph-mon[131105]: pgmap v568: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-17T00:07:42.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:41 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1224042637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-17T00:07:42.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.446+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcf04067b00 con 0x7fcf04073550 2023-11-17T00:07:42.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.447+0000 7fcf00ff9700 1 -- 172.21.15.23:0/1748232693 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcef409b270 con 0x7fcf04073550 2023-11-17T00:07:42.448 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 msgr2=0x7fceec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 0x7fceec0a3600 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcf04074850 tx=0x7fcef800b410 comp rx=0 tx=0).stop 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 msgr2=0x7fcf041982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b680 tx=0x7fcef4005740 comp rx=0 tx=0).stop 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 shutdown_connections 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf04108190 0x7fcf0419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf04073550 0x7fcf041982a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.449+0000 7fcf09bff700 1 --2- 172.21.15.23:0/1748232693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf04074b30 0x7fcf041987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.450+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 >> 172.21.15.23:0/1748232693 conn(0x7fcf040fb8b0 msgr2=0x7fcf040fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:42.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.450+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 shutdown_connections 2023-11-17T00:07:42.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.450+0000 7fcf09bff700 1 -- 172.21.15.23:0/1748232693 wait complete. 2023-11-17T00:07:42.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-17T00:07:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 -- 172.21.15.23:0/2594659428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 msgr2=0x7f41ec100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 --2- 172.21.15.23:0/2594659428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec100640 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009990 tx=0x7f41e8009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 -- 172.21.15.23:0/2594659428 shutdown_connections 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 --2- 172.21.15.23:0/2594659428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101db0 0x7f41ec108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 --2- 172.21.15.23:0/2594659428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec100640 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 --2- 172.21.15.23:0/2594659428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101420 0x7f41ec101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 -- 172.21.15.23:0/2594659428 >> 172.21.15.23:0/2594659428 conn(0x7f41ec0fb810 msgr2=0x7f41ec0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 -- 172.21.15.23:0/2594659428 shutdown_connections 2023-11-17T00:07:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.676+0000 7f41f43ff700 1 -- 172.21.15.23:0/2594659428 wait complete. 2023-11-17T00:07:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.677+0000 7f41f43ff700 1 Processor -- start 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.677+0000 7f41f43ff700 1 -- start start 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101420 0x7f41ec196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101db0 0x7f41ec19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19db60 con 0x7f41ec100230 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19dca0 con 0x7f41ec101db0 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f43ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19de30 con 0x7f41ec101420 2023-11-17T00:07:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47982/0 (socket says 172.21.15.23:47982) 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 -- 172.21.15.23:0/1602539896 learned_addr learned my addr 172.21.15.23:0/1602539896 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f199a700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101420 0x7f41ec196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f299c700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101db0 0x7f41ec19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 -- 172.21.15.23:0/1602539896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101420 msgr2=0x7f41ec196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101420 0x7f41ec196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 -- 172.21.15.23:0/1602539896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101db0 msgr2=0x7f41ec19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101db0 0x7f41ec19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.678+0000 7f41f219b700 1 -- 172.21.15.23:0/1602539896 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80095f0 con 0x7f41ec100230 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41f219b700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f41e8004730 tx=0x7f41e8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e801d070 con 0x7f41ec100230 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec10bbe0 con 0x7f41ec100230 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41e8021410 con 0x7f41ec100230 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8017650 con 0x7f41ec100230 2023-11-17T00:07:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.679+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec10ba50 con 0x7f41ec100230 2023-11-17T00:07:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.680+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec1077d0 con 0x7f41ec100230 2023-11-17T00:07:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.682+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f41e8021580 con 0x7f41ec100230 2023-11-17T00:07:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.683+0000 7f41e37fe700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 0x7f41d80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.683+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f41e80ce6b0 con 0x7f41ec100230 2023-11-17T00:07:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.684+0000 7f41f199a700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 0x7f41d80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.684+0000 7f41f199a700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 0x7f41d80a3680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009c60 tx=0x7f41dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.685+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f41e8097ae0 con 0x7f41ec100230 2023-11-17T00:07:42.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.917+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f41ec061700 con 0x7f41ec100230 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.917+0000 7f41e37fe700 1 -- 172.21.15.23:0/1602539896 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) v1 ==== 81+0+591 (secure 0 0 0) 0x7f41e809b100 con 0x7f41ec100230 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2730190791 2023-11-17T23:51:30.957768+0000 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3289144239 2023-11-17T23:51:30.957768+0000 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4236499031 2023-11-17T23:51:04.390538+0000 2023-11-17T00:07:42.919 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-17T00:07:42.920 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2789478528 2023-11-17T23:51:04.390538+0000 2023-11-17T00:07:42.920 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3874168431 2023-11-17T23:51:30.957768+0000 2023-11-17T00:07:42.920 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1477838149 2023-11-17T23:51:04.390538+0000 2023-11-17T00:07:42.920 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/792254178 2023-11-17T23:51:04.390538+0000 2023-11-17T00:07:42.920 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1353533979 2023-11-17T23:51:30.957768+0000 2023-11-17T00:07:42.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 msgr2=0x7f41d80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 0x7f41d80a3680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009c60 tx=0x7f41dc009260 comp rx=0 tx=0).stop 2023-11-17T00:07:42.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 msgr2=0x7f41ec195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:42.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f41e8004730 tx=0x7f41e8004760 comp rx=0 tx=0).stop 2023-11-17T00:07:42.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 shutdown_connections 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41ec101db0 0x7f41ec19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41ec100230 0x7f41ec195fe0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f41d80a11d0 0x7f41d80a3680 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 --2- 172.21.15.23:0/1602539896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41ec101420 0x7f41ec196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 >> 172.21.15.23:0/1602539896 conn(0x7f41ec0fb810 msgr2=0x7f41ec104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 shutdown_connections 2023-11-17T00:07:42.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:42.920+0000 7f41f43ff700 1 -- 172.21.15.23:0/1602539896 wait complete. 2023-11-17T00:07:42.923 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2023-11-17T00:07:42.941 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-17T00:07:42.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:07:42.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-17T00:07:43.141 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:07:43.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:07:43.142 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-17T00:07:43.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:07:43.268 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-11-17T00:07:43.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:42 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3638711517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:42 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1748232693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:42 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3638711517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:42 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1748232693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.357 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:42 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3638711517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:42 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1748232693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-17T00:07:43.365 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-17T00:07:43.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:07:43.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-17T00:07:43.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-17T00:07:43.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2023-11-17T00:07:43.550 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-17T00:07:43.561 INFO:tasks.cephadm:Teardown begin 2023-11-17T00:07:43.561 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-17T00:07:43.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:07:43.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 -- 172.21.15.23:0/1173761124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101480 msgr2=0x7f7b1c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 --2- 172.21.15.23:0/1173761124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101480 0x7f7b1c1018d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009990 tx=0x7f7b0c009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 -- 172.21.15.23:0/1173761124 shutdown_connections 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 --2- 172.21.15.23:0/1173761124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101e10 0x7f7b1c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 --2- 172.21.15.23:0/1173761124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 --2- 172.21.15.23:0/1173761124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101480 0x7f7b1c1018d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 -- 172.21.15.23:0/1173761124 >> 172.21.15.23:0/1173761124 conn(0x7f7b1c0fb870 msgr2=0x7f7b1c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:43.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 -- 172.21.15.23:0/1173761124 shutdown_connections 2023-11-17T00:07:43.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.783+0000 7f7b23bff700 1 -- 172.21.15.23:0/1173761124 wait complete. 2023-11-17T00:07:43.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 Processor -- start 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 -- start start 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 0x7f7b1c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c19fd50 con 0x7f7b1c101e10 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c19fe90 con 0x7f7b1c100290 2023-11-17T00:07:43.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b23bff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c1a0020 con 0x7f7b1c101480 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b2119a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b2219c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 0x7f7b1c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b2119a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:33256/0 (socket says 172.21.15.23:33256) 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.785+0000 7f7b2119a700 1 -- 172.21.15.23:0/2617093597 learned_addr learned my addr 172.21.15.23:0/2617093597 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2199b700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 -- 172.21.15.23:0/2617093597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 msgr2=0x7f7b1c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 -- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 msgr2=0x7f7b1c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:43.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 0x7f7b1c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 -- 172.21.15.23:0/2617093597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c0095f0 con 0x7f7b1c101480 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2199b700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2219c700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 0x7f7b1c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b2119a700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00b620 tx=0x7f7b0c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c01d070 con 0x7f7b1c101480 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b0c022470 con 0x7f7b1c101480 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c017650 con 0x7f7b1c101480 2023-11-17T00:07:43.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.786+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c10bc50 con 0x7f7b1c101480 2023-11-17T00:07:43.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.787+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c10bac0 con 0x7f7b1c101480 2023-11-17T00:07:43.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.788+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c1928e0 con 0x7f7b1c101480 2023-11-17T00:07:43.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.790+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7b0c0225e0 con 0x7f7b1c101480 2023-11-17T00:07:43.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.791+0000 7f7b12ffd700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 0x7f7b080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:43.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.791+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f7b0c0ce2a0 con 0x7f7b1c101480 2023-11-17T00:07:43.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.791+0000 7f7b2199b700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 0x7f7b080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:43.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.792+0000 7f7b2199b700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 0x7f7b080a3680 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c1012e0 tx=0x7f7b18009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:43.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:43.793+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7b0c0976d0 con 0x7f7b1c101480 2023-11-17T00:07:44.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:43 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1602539896' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-17T00:07:44.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:43 smithi023 ceph-mon[97824]: pgmap v569: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-11-17T00:07:44.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.053+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b1c02cfe0 con 0x7f7b1c101480 2023-11-17T00:07:44.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.054+0000 7f7b12ffd700 1 -- 172.21.15.23:0/2617093597 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10824 (secure 0 0 0) 0x7f7b0c09acf0 con 0x7f7b1c101480 2023-11-17T00:07:44.058 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:07:44.058 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":25681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2948132470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2948132470},{"type":"v1","addr":"172.21.15.173:6837","nonce":2948132470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-17T00:05:52.253173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25708":{"gid":25708,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2831566569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2831566569},{"type":"v1","addr":"172.21.15.173:6833","nonce":2831566569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:07:44.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 msgr2=0x7f7b080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 0x7f7b080a3680 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c1012e0 tx=0x7f7b18009260 comp rx=0 tx=0).stop 2023-11-17T00:07:44.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 msgr2=0x7f7b1c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00b620 tx=0x7f7b0c005740 comp rx=0 tx=0).stop 2023-11-17T00:07:44.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 shutdown_connections 2023-11-17T00:07:44.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b1c101e10 0x7f7b1c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b1c100290 0x7f7b1c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f7b080a11d0 0x7f7b080a3680 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 --2- 172.21.15.23:0/2617093597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b1c101480 0x7f7b1c198710 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 >> 172.21.15.23:0/2617093597 conn(0x7f7b1c0fb870 msgr2=0x7f7b1c106dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:44.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 shutdown_connections 2023-11-17T00:07:44.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.057+0000 7f7b23bff700 1 -- 172.21.15.23:0/2617093597 wait complete. 2023-11-17T00:07:44.061 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 33 2023-11-17T00:07:44.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-17T00:07:44.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 -- 172.21.15.23:0/2269777631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 msgr2=0x7f97b01006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 --2- 172.21.15.23:0/2269777631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01006c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009990 tx=0x7f97a4009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 -- 172.21.15.23:0/2269777631 shutdown_connections 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 --2- 172.21.15.23:0/2269777631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b01086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 --2- 172.21.15.23:0/2269777631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01006c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 --2- 172.21.15.23:0/2269777631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 0x7f97b01018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 -- 172.21.15.23:0/2269777631 >> 172.21.15.23:0/2269777631 conn(0x7f97b00fb890 msgr2=0x7f97b00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 -- 172.21.15.23:0/2269777631 shutdown_connections 2023-11-17T00:07:44.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.284+0000 7f97b71ff700 1 -- 172.21.15.23:0/2269777631 wait complete. 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.285+0000 7f97b71ff700 1 Processor -- start 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.285+0000 7f97b71ff700 1 -- start start 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 0x7f97b0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b019fd60 con 0x7f97b01002b0 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b019fea0 con 0x7f97b01014a0 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b71ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b01a0030 con 0x7f97b0101e30 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b4f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b4f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48026/0 (socket says 172.21.15.23:48026) 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b4f9b700 1 -- 172.21.15.23:0/3970169869 learned_addr learned my addr 172.21.15.23:0/3970169869 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97affff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 0x7f97b0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 -- 172.21.15.23:0/3970169869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 msgr2=0x7f97b0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 0x7f97b0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 -- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 msgr2=0x7f97b01981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.286+0000 7f97b579c700 1 -- 172.21.15.23:0/3970169869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a40095f0 con 0x7f97b0101e30 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97b579c700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b019f790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f97a800efc0 tx=0x7f97a800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8018070 con 0x7f97b0101e30 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97a800f040 con 0x7f97b0101e30 2023-11-17T00:07:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800cd40 con 0x7f97b0101e30 2023-11-17T00:07:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b010bc80 con 0x7f97b0101e30 2023-11-17T00:07:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.287+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b010baf0 con 0x7f97b0101e30 2023-11-17T00:07:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.288+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b01a0670 con 0x7f97b0101e30 2023-11-17T00:07:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.290+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f97a8003a00 con 0x7f97b0101e30 2023-11-17T00:07:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.292+0000 7f97adffb700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 0x7f97a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.292+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f97a80cc8d0 con 0x7f97b0101e30 2023-11-17T00:07:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.292+0000 7f97b4f9b700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 0x7f97a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.292+0000 7f97b4f9b700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 0x7f97a00a3680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f97a400b2e0 tx=0x7f97a4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:44.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.293+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f97a8095d00 con 0x7f97b0101e30 2023-11-17T00:07:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:43 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1602539896' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-17T00:07:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:43 smithi173 ceph-mon[131534]: pgmap v569: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-11-17T00:07:44.361 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:43 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1602539896' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-17T00:07:44.361 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:43 smithi156 ceph-mon[131105]: pgmap v569: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-11-17T00:07:44.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.548+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97b01a0c20 con 0x7f97b0101e30 2023-11-17T00:07:44.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.549+0000 7f97adffb700 1 -- 172.21.15.23:0/3970169869 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10824 (secure 0 0 0) 0x7f97a8099320 con 0x7f97b0101e30 2023-11-17T00:07:44.550 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:07:44.551 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14463,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1622872783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1622872783},{"type":"v1","addr":"172.21.15.23:6835","nonce":1622872783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4279927842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4279927842},{"type":"v1","addr":"172.21.15.23:6837","nonce":4279927842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14529,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/3704661789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":3704661789},{"type":"v1","addr":"172.21.15.23:6839","nonce":3704661789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14541,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/3035227424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":3035227424},{"type":"v1","addr":"172.21.15.23:6841","nonce":3035227424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2227062426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2227062426},{"type":"v1","addr":"172.21.15.156:6833","nonce":2227062426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":25681,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/2948132470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":2948132470},{"type":"v1","addr":"172.21.15.173:6837","nonce":2948132470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T23:58:46.253225+0000","modified":"2023-11-17T00:05:52.253173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24565,"mds_2":24529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/3543343885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":3543343885},{"type":"v1","addr":"172.21.15.156:6835","nonce":3543343885}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24529":{"gid":24529,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":19,"addr":"172.21.15.173:6835/405519546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":405519546},{"type":"v1","addr":"172.21.15.173:6835","nonce":405519546}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/180108175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":180108175},{"type":"v1","addr":"172.21.15.156:6837","nonce":180108175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24565":{"gid":24565,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.173:6839/3306732798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":3306732798},{"type":"v1","addr":"172.21.15.173:6839","nonce":3306732798}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1637342449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1637342449},{"type":"v1","addr":"172.21.15.156:6839","nonce":1637342449}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25708":{"gid":25708,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2831566569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2831566569},{"type":"v1","addr":"172.21.15.173:6833","nonce":2831566569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.551+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 msgr2=0x7f97a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.551+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 0x7f97a00a3680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f97a400b2e0 tx=0x7f97a4005fb0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.551+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 msgr2=0x7f97b019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b019f790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f97a800efc0 tx=0x7f97a800c3a0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 shutdown_connections 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f97a00a11d0 0x7f97a00a3680 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97b0101e30 0x7f97b019f790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97b01002b0 0x7f97b01981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 --2- 172.21.15.23:0/3970169869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b01014a0 0x7f97b0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 >> 172.21.15.23:0/3970169869 conn(0x7f97b00fb890 msgr2=0x7f97b00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 shutdown_connections 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.552+0000 7f97b71ff700 1 -- 172.21.15.23:0/3970169869 wait complete. 2023-11-17T00:07:44.554 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 33 2023-11-17T00:07:44.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-17T00:07:44.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 -- 172.21.15.23:0/2514520102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 msgr2=0x7fb0501086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 --2- 172.21.15.23:0/2514520102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb0501086d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb048009f90 tx=0x7fb04800b330 comp rx=0 tx=0).stop 2023-11-17T00:07:44.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 -- 172.21.15.23:0/2514520102 shutdown_connections 2023-11-17T00:07:44.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 --2- 172.21.15.23:0/2514520102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb0501086d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 --2- 172.21.15.23:0/2514520102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050100290 0x7fb0501006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 --2- 172.21.15.23:0/2514520102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050101480 0x7fb0501018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 -- 172.21.15.23:0/2514520102 >> 172.21.15.23:0/2514520102 conn(0x7fb0500fb870 msgr2=0x7fb0500fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:44.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 -- 172.21.15.23:0/2514520102 shutdown_connections 2023-11-17T00:07:44.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.775+0000 7fb0553ff700 1 -- 172.21.15.23:0/2514520102 wait complete. 2023-11-17T00:07:44.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 Processor -- start 2023-11-17T00:07:44.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 -- start start 2023-11-17T00:07:44.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 0x7fb050198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb05019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05019fc00 con 0x7fb050101480 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05019fd40 con 0x7fb050100290 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.776+0000 7fb0553ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05019fed0 con 0x7fb050101e10 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48040/0 (socket says 172.21.15.23:48040) 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 -- 172.21.15.23:0/652761769 learned_addr learned my addr 172.21.15.23:0/652761769 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:44.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04effd700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 0x7fb050198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04f7fe700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb05019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 -- 172.21.15.23:0/652761769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 msgr2=0x7fb05019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb05019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 -- 172.21.15.23:0/652761769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 msgr2=0x7fb050198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 0x7fb050198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 -- 172.21.15.23:0/652761769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044009520 con 0x7fb050101480 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04effd700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 0x7fb050198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:44.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.777+0000 7fb04e7fc700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb04400eb30 tx=0x7fb04400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:44.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.778+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044009880 con 0x7fb050101480 2023-11-17T00:07:44.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.778+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0440044e0 con 0x7fb050101480 2023-11-17T00:07:44.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.778+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb048009c40 con 0x7fb050101480 2023-11-17T00:07:44.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.778+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044003d80 con 0x7fb050101480 2023-11-17T00:07:44.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.778+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05010bb90 con 0x7fb050101480 2023-11-17T00:07:44.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.779+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb050067a60 con 0x7fb050101480 2023-11-17T00:07:44.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.781+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb044003ee0 con 0x7fb050101480 2023-11-17T00:07:44.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.783+0000 7fb037fff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 0x7fb0380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:44.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.783+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fb044014070 con 0x7fb050101480 2023-11-17T00:07:44.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.783+0000 7fb04effd700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 0x7fb0380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:44.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.784+0000 7fb04effd700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 0x7fb0380a3600 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb0501012e0 tx=0x7fb040009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:44.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.787+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb044096430 con 0x7fb050101480 2023-11-17T00:07:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:44.999+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0501a09f0 con 0x7fb050101480 2023-11-17T00:07:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.001+0000 7fb037fff700 1 -- 172.21.15.23:0/652761769 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24603 (secure 0 0 0) 0x7fb044016330 con 0x7fb050101480 2023-11-17T00:07:45.002 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-11-17T00:07:45.003 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"d04064ea-84da-11ee-8dbe-212e2dc638e7","created":"2023-11-16T23:50:32.514395+0000","modified":"2023-11-17T00:05:51.321983+0000","last_up_change":"2023-11-16T23:57:56.685581+0000","last_in_change":"2023-11-16T23:57:38.148714+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T23:54:03.596917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T23:58:44.557031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T23:58:45.676734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T23:58:49.239193+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a823ad49-73e7-442f-89dc-91875e102c7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6803","nonce":2230726464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6805","nonce":2230726464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6809","nonce":2230726464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2230726464},{"type":"v1","addr":"172.21.15.23:6807","nonce":2230726464}]},"public_addr":"172.21.15.23:6803/2230726464","cluster_addr":"172.21.15.23:6805/2230726464","heartbeat_back_addr":"172.21.15.23:6809/2230726464","heartbeat_front_addr":"172.21.15.23:6807/2230726464","state":["exists","up"]},{"osd":1,"uuid":"26bd38f3-3da3-45e1-9d3b-d375d7fa95ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6801","nonce":2641693166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6803","nonce":2641693166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6807","nonce":2641693166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2641693166},{"type":"v1","addr":"172.21.15.156:6805","nonce":2641693166}]},"public_addr":"172.21.15.156:6801/2641693166","cluster_addr":"172.21.15.156:6803/2641693166","heartbeat_back_addr":"172.21.15.156:6807/2641693166","heartbeat_front_addr":"172.21.15.156:6805/2641693166","state":["exists","up"]},{"osd":2,"uuid":"6b22c974-66e4-455c-882c-4804d393aec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6801","nonce":2331881016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6803","nonce":2331881016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6807","nonce":2331881016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2331881016},{"type":"v1","addr":"172.21.15.173:6805","nonce":2331881016}]},"public_addr":"172.21.15.173:6801/2331881016","cluster_addr":"172.21.15.173:6803/2331881016","heartbeat_back_addr":"172.21.15.173:6807/2331881016","heartbeat_front_addr":"172.21.15.173:6805/2331881016","state":["exists","up"]},{"osd":3,"uuid":"82122923-cabb-475e-ab26-d6a7bb6c88ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6811","nonce":2975279435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6813","nonce":2975279435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6817","nonce":2975279435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2975279435},{"type":"v1","addr":"172.21.15.23:6815","nonce":2975279435}]},"public_addr":"172.21.15.23:6811/2975279435","cluster_addr":"172.21.15.23:6813/2975279435","heartbeat_back_addr":"172.21.15.23:6817/2975279435","heartbeat_front_addr":"172.21.15.23:6815/2975279435","state":["exists","up"]},{"osd":4,"uuid":"dbe06115-7a9f-4be5-b2df-3ac182c2f34f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6809","nonce":1251331941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6811","nonce":1251331941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6815","nonce":1251331941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1251331941},{"type":"v1","addr":"172.21.15.156:6813","nonce":1251331941}]},"public_addr":"172.21.15.156:6809/1251331941","cluster_addr":"172.21.15.156:6811/1251331941","heartbeat_back_addr":"172.21.15.156:6815/1251331941","heartbeat_front_addr":"172.21.15.156:6813/1251331941","state":["exists","up"]},{"osd":5,"uuid":"5cb74da1-90da-4bfa-a528-b08833cabdcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6809","nonce":2003227245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6811","nonce":2003227245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6815","nonce":2003227245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2003227245},{"type":"v1","addr":"172.21.15.173:6813","nonce":2003227245}]},"public_addr":"172.21.15.173:6809/2003227245","cluster_addr":"172.21.15.173:6811/2003227245","heartbeat_back_addr":"172.21.15.173:6815/2003227245","heartbeat_front_addr":"172.21.15.173:6813/2003227245","state":["exists","up"]},{"osd":6,"uuid":"37b75c4c-cc8a-4327-bfda-9da89338654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6819","nonce":1429107943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6821","nonce":1429107943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6825","nonce":1429107943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1429107943},{"type":"v1","addr":"172.21.15.23:6823","nonce":1429107943}]},"public_addr":"172.21.15.23:6819/1429107943","cluster_addr":"172.21.15.23:6821/1429107943","heartbeat_back_addr":"172.21.15.23:6825/1429107943","heartbeat_front_addr":"172.21.15.23:6823/1429107943","state":["exists","up"]},{"osd":7,"uuid":"1f9a70dc-a716-42aa-a798-635dd1f18ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6817","nonce":2873383539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6819","nonce":2873383539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6823","nonce":2873383539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2873383539},{"type":"v1","addr":"172.21.15.156:6821","nonce":2873383539}]},"public_addr":"172.21.15.156:6817/2873383539","cluster_addr":"172.21.15.156:6819/2873383539","heartbeat_back_addr":"172.21.15.156:6823/2873383539","heartbeat_front_addr":"172.21.15.156:6821/2873383539","state":["exists","up"]},{"osd":8,"uuid":"31074fdd-3ccc-460e-91e1-10339bf34c95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6817","nonce":2703397724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6819","nonce":2703397724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6823","nonce":2703397724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2703397724},{"type":"v1","addr":"172.21.15.173:6821","nonce":2703397724}]},"public_addr":"172.21.15.173:6817/2703397724","cluster_addr":"172.21.15.173:6819/2703397724","heartbeat_back_addr":"172.21.15.173:6823/2703397724","heartbeat_front_addr":"172.21.15.173:6821/2703397724","state":["exists","up"]},{"osd":9,"uuid":"87031132-5932-4aa2-ad9d-4b706edfac8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6827","nonce":2722703119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6829","nonce":2722703119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6833","nonce":2722703119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2722703119},{"type":"v1","addr":"172.21.15.23:6831","nonce":2722703119}]},"public_addr":"172.21.15.23:6827/2722703119","cluster_addr":"172.21.15.23:6829/2722703119","heartbeat_back_addr":"172.21.15.23:6833/2722703119","heartbeat_front_addr":"172.21.15.23:6831/2722703119","state":["exists","up"]},{"osd":10,"uuid":"afa77aab-17f7-4ab5-a277-c10102ad7e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6825","nonce":651591322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6827","nonce":651591322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6831","nonce":651591322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":651591322},{"type":"v1","addr":"172.21.15.156:6829","nonce":651591322}]},"public_addr":"172.21.15.156:6825/651591322","cluster_addr":"172.21.15.156:6827/651591322","heartbeat_back_addr":"172.21.15.156:6831/651591322","heartbeat_front_addr":"172.21.15.156:6829/651591322","state":["exists","up"]},{"osd":11,"uuid":"f38c6af8-9481-4757-9973-dba7aa1a03b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":79,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6825","nonce":3375444046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6827","nonce":3375444046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6831","nonce":3375444046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3375444046},{"type":"v1","addr":"172.21.15.173:6829","nonce":3375444046}]},"public_addr":"172.21.15.173:6825/3375444046","cluster_addr":"172.21.15.173:6827/3375444046","heartbeat_back_addr":"172.21.15.173:6831/3375444046","heartbeat_front_addr":"172.21.15.173:6829/3375444046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:15.396649+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:53:38.219287+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:00.348923+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:27.067311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:54:58.732557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:21.049861+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:55:48.035809+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:10.780231+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:32.619800+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:56:59.292266+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:32.170337+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T23:57:54.828509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2730190791":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6800/3289144239":"2023-11-17T23:51:30.957768+0000","172.21.15.23:0/4236499031":"2023-11-17T23:51:04.390538+0000","172.21.15.23:6800/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/2789478528":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/3874168431":"2023-11-17T23:51:30.957768+0000","172.21.15.23:6801/1477838149":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/792254178":"2023-11-17T23:51:04.390538+0000","172.21.15.23:0/1353533979":"2023-11-17T23:51:30.957768+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T00:07:45.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 msgr2=0x7fb0380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 0x7fb0380a3600 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb0501012e0 tx=0x7fb040009310 comp rx=0 tx=0).stop 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 msgr2=0x7fb0501985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb04400eb30 tx=0x7fb04400ee40 comp rx=0 tx=0).stop 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 shutdown_connections 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fb0380a1150 0x7fb0380a3600 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb050101e10 0x7fb05019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb050100290 0x7fb050198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 --2- 172.21.15.23:0/652761769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb050101480 0x7fb0501985c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.003+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 >> 172.21.15.23:0/652761769 conn(0x7fb0500fb870 msgr2=0x7fb0500fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.004+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 shutdown_connections 2023-11-17T00:07:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.004+0000 7fb0553ff700 1 -- 172.21.15.23:0/652761769 wait complete. 2023-11-17T00:07:45.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-17T00:07:45.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 -- 172.21.15.23:0/3153425769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 msgr2=0x7f35e0108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 --2- 172.21.15.23:0/3153425769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e0108670 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b0a0 tx=0x7f35d800b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 -- 172.21.15.23:0/3153425769 shutdown_connections 2023-11-17T00:07:45.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 --2- 172.21.15.23:0/3153425769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e0108670 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 --2- 172.21.15.23:0/3153425769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e0100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 --2- 172.21.15.23:0/3153425769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 0x7f35e0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 -- 172.21.15.23:0/3153425769 >> 172.21.15.23:0/3153425769 conn(0x7f35e00fb810 msgr2=0x7f35e00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:45.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 -- 172.21.15.23:0/3153425769 shutdown_connections 2023-11-17T00:07:45.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.231+0000 7f35e73ff700 1 -- 172.21.15.23:0/3153425769 wait complete. 2023-11-17T00:07:45.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.232+0000 7f35e73ff700 1 Processor -- start 2023-11-17T00:07:45.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.232+0000 7f35e73ff700 1 -- start start 2023-11-17T00:07:45.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 0x7f35e01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e019fe30 con 0x7f35e0101db0 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e019ff70 con 0x7f35e0100230 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e73ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01a0100 con 0x7f35e0101420 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48052/0 (socket says 172.21.15.23:48052) 2023-11-17T00:07:45.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e499a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 0x7f35e01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 -- 172.21.15.23:0/4110558949 learned_addr learned my addr 172.21.15.23:0/4110558949 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e519b700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 -- 172.21.15.23:0/4110558949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 msgr2=0x7f35e01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 0x7f35e01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 -- 172.21.15.23:0/4110558949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 msgr2=0x7f35e01982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e01982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e599c700 1 -- 172.21.15.23:0/4110558949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8009d00 con 0x7f35e0101db0 2023-11-17T00:07:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.233+0000 7f35e519b700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35e599c700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f35d800d010 tx=0x7f35d800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d800e040 con 0x7f35e0101db0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35d8003e10 con 0x7f35e0101db0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d80052a0 con 0x7f35e0101db0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e010bc00 con 0x7f35e0101db0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.234+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e010ba70 con 0x7f35e0101db0 2023-11-17T00:07:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.235+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e010bd90 con 0x7f35e0101db0 2023-11-17T00:07:45.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.238+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f35d802e020 con 0x7f35e0101db0 2023-11-17T00:07:45.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.240+0000 7f35d67fc700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 0x7f35cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:45.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.240+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f35d80ce160 con 0x7f35e0101db0 2023-11-17T00:07:45.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.240+0000 7f35e519b700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 0x7f35cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:45.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.241+0000 7f35e519b700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 0x7f35cc0a3600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f35e0101280 tx=0x7f35dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:45.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.242+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f35d8097590 con 0x7f35e0101db0 2023-11-17T00:07:45.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:44 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2617093597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:44 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3970169869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:44 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2617093597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:44 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3970169869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:44 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2617093597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.365 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:44 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3970169869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-17T00:07:45.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.500+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f35e01a0bb0 con 0x7f35e0101db0 2023-11-17T00:07:45.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.929+0000 7f35d67fc700 1 -- 172.21.15.23:0/4110558949 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f35d8095e60 con 0x7f35e0101db0 2023-11-17T00:07:45.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 msgr2=0x7f35cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 0x7f35cc0a3600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f35e0101280 tx=0x7f35dc009260 comp rx=0 tx=0).stop 2023-11-17T00:07:45.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 msgr2=0x7f35e019f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:45.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f35d800d010 tx=0x7f35d800bc80 comp rx=0 tx=0).stop 2023-11-17T00:07:45.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 shutdown_connections 2023-11-17T00:07:45.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35e0101db0 0x7f35e019f860 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35e0100230 0x7f35e01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.931+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f35cc0a1150 0x7f35cc0a3600 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.932+0000 7f35e73ff700 1 --2- 172.21.15.23:0/4110558949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35e0101420 0x7f35e01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:45.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.932+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 >> 172.21.15.23:0/4110558949 conn(0x7f35e00fb810 msgr2=0x7f35e0106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:45.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.932+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 shutdown_connections 2023-11-17T00:07:45.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:45.932+0000 7f35e73ff700 1 -- 172.21.15.23:0/4110558949 wait complete. 2023-11-17T00:07:45.939 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-17T00:07:45.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-17T00:07:46.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/652761769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mon[131534]: pgmap v570: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2023-11-17T00:07:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-17T00:07:46.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mon[131534]: osdmap e94: 12 total, 12 up, 12 in 2023-11-17T00:07:46.109 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: 2023-11-17T00:07:45.806+0000 7fc13d0d8700 -1 MDSIOContextBase: failed with -108, restarting... 2023-11-17T00:07:46.109 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: -1> 2023-11-17T00:07:45.806+0000 7fc13d0d8700 -1 MDSIOContextBase: failed with -108, restarting... 2023-11-17T00:07:46.110 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.110 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.110 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mds[157040]: -- 172.21.15.173:0/1588206794 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a89e463520 con 0x55a89e438800 2023-11-17T00:07:46.110 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: starting mds.f at 2023-11-17T00:07:46.110 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mds[157722]: -- 172.21.15.173:0/1241715799 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561ac62d5520 con 0x561ac62ab000 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: starting mds.i at 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.111 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-mds[160681]: -- 172.21.15.173:0/3747947290 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f62e07b520 con 0x55f62e050c00 2023-11-17T00:07:46.112 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:07:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: starting mds.l at 2023-11-17T00:07:46.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 -- 172.21.15.23:0/1854697310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64281014b0 msgr2=0x7f6428101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:46.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 --2- 172.21.15.23:0/1854697310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64281014b0 0x7f6428101900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6418009990 tx=0x7f6418009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 -- 172.21.15.23:0/1854697310 shutdown_connections 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 --2- 172.21.15.23:0/1854697310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6428101e40 0x7f6428108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 --2- 172.21.15.23:0/1854697310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281002c0 0x7f64281006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 --2- 172.21.15.23:0/1854697310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64281014b0 0x7f6428101900 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 -- 172.21.15.23:0/1854697310 >> 172.21.15.23:0/1854697310 conn(0x7f64280fb8a0 msgr2=0x7f64280fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 -- 172.21.15.23:0/1854697310 shutdown_connections 2023-11-17T00:07:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.165+0000 7f642f7ff700 1 -- 172.21.15.23:0/1854697310 wait complete. 2023-11-17T00:07:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 Processor -- start 2023-11-17T00:07:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 -- start start 2023-11-17T00:07:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64281002c0 0x7f6428198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 0x7f64281987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642819fde0 con 0x7f6428101e40 2023-11-17T00:07:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642819ff20 con 0x7f64281014b0 2023-11-17T00:07:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.166+0000 7f642f7ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64281a00b0 con 0x7f64281002c0 2023-11-17T00:07:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48078/0 (socket says 172.21.15.23:48078) 2023-11-17T00:07:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 -- 172.21.15.23:0/3985628 learned_addr learned my addr 172.21.15.23:0/3985628 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642cd9a700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 0x7f64281987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642d59b700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64281002c0 0x7f6428198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 -- 172.21.15.23:0/3985628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64281002c0 msgr2=0x7f6428198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64281002c0 0x7f6428198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 -- 172.21.15.23:0/3985628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 msgr2=0x7f64281987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 0x7f64281987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 -- 172.21.15.23:0/3985628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420009c40 con 0x7f6428101e40 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642cd9a700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 0x7f64281987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642dd9c700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f642000c370 tx=0x7f642000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420017070 con 0x7f6428101e40 2023-11-17T00:07:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f642000f040 con 0x7f6428101e40 2023-11-17T00:07:46.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642000e050 con 0x7f6428101e40 2023-11-17T00:07:46.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.167+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64180095f0 con 0x7f6428101e40 2023-11-17T00:07:46.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.168+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642810bc70 con 0x7f6428101e40 2023-11-17T00:07:46.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.169+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6428067b00 con 0x7f6428101e40 2023-11-17T00:07:46.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.171+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6420004520 con 0x7f6428101e40 2023-11-17T00:07:46.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.172+0000 7f641e7fc700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 0x7f64140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.172+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9336+0+0 (secure 0 0 0) 0x7f64200cc6d0 con 0x7f6428101e40 2023-11-17T00:07:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.173+0000 7f642d59b700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 0x7f64140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.173+0000 7f642d59b700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 0x7f64140a3600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6428101310 tx=0x7f6424009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.176+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f64200959d0 con 0x7f6428101e40 2023-11-17T00:07:46.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a[97800]: 2023-11-17T00:07:45.921+0000 7f6f2917e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T00:07:46.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:45 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/652761769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:46.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:45 smithi023 ceph-mon[97824]: pgmap v570: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2023-11-17T00:07:46.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:45 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-17T00:07:46.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:45 smithi023 ceph-mon[97824]: osdmap e94: 12 total, 12 up, 12 in 2023-11-17T00:07:46.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.358 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.358 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mds[156453]: -- 172.21.15.156:0/515040121 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c7445b520 con 0x556c74430c00 2023-11-17T00:07:46.358 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: starting mds.b at 2023-11-17T00:07:46.359 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.359 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.359 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mds[155772]: -- 172.21.15.156:0/607694142 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9ec607520 con 0x55b9ec5dd000 2023-11-17T00:07:46.359 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: starting mds.e at 2023-11-17T00:07:46.359 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: ignoring --setuser ceph since I am not root 2023-11-17T00:07:46.360 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: ignoring --setgroup ceph since I am not root 2023-11-17T00:07:46.360 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: starting mds.h at 2023-11-17T00:07:46.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/652761769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T00:07:46.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mon[131105]: pgmap v570: 129 pgs: 129 active+clean; 311 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 8 op/s 2023-11-17T00:07:46.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-17T00:07:46.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:45 smithi156 ceph-mon[131105]: osdmap e94: 12 total, 12 up, 12 in 2023-11-17T00:07:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.431+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f64281a0c00 con 0x7f6428101e40 2023-11-17T00:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.932+0000 7f641e7fc700 1 -- 172.21.15.23:0/3985628 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6420098ff0 con 0x7f6428101e40 2023-11-17T00:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.934+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 msgr2=0x7f64140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 0x7f64140a3600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6428101310 tx=0x7f6424009260 comp rx=0 tx=0).stop 2023-11-17T00:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 msgr2=0x7f642819f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f642000c370 tx=0x7f642000c680 comp rx=0 tx=0).stop 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 shutdown_connections 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6428101e40 0x7f642819f810 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f64140a1150 0x7f64140a3600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64281002c0 0x7f6428198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 --2- 172.21.15.23:0/3985628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64281014b0 0x7f64281987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 >> 172.21.15.23:0/3985628 conn(0x7f64280fb8a0 msgr2=0x7f64280fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 shutdown_connections 2023-11-17T00:07:46.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:46.935+0000 7f642f7ff700 1 -- 172.21.15.23:0/3985628 wait complete. 2023-11-17T00:07:46.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-17T00:07:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 -- 172.21.15.23:0/2445538924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc101e50 msgr2=0x7fabdc108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2445538924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc101e50 0x7fabdc108710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fabd400b0a0 tx=0x7fabd400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 -- 172.21.15.23:0/2445538924 shutdown_connections 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2445538924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc101e50 0x7fabdc108710 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2445538924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc1002d0 0x7fabdc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2445538924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1014c0 0x7fabdc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 -- 172.21.15.23:0/2445538924 >> 172.21.15.23:0/2445538924 conn(0x7fabdc0fb8b0 msgr2=0x7fabdc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 -- 172.21.15.23:0/2445538924 shutdown_connections 2023-11-17T00:07:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.167+0000 7fabe43ff700 1 -- 172.21.15.23:0/2445538924 wait complete. 2023-11-17T00:07:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.168+0000 7fabe43ff700 1 Processor -- start 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 -- start start 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 0x7fabdc198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc101e50 0x7fabdc19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc19fef0 con 0x7fabdc1014c0 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1a0030 con 0x7fabdc1002d0 2023-11-17T00:07:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe43ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1a01c0 con 0x7fabdc101e50 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe199a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 0x7fabdc198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe199a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48096/0 (socket says 172.21.15.23:48096) 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe199a700 1 -- 172.21.15.23:0/2738975911 learned_addr learned my addr 172.21.15.23:0/2738975911 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.169+0000 7fabe299c700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc101e50 0x7fabdc19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 -- 172.21.15.23:0/2738975911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc101e50 msgr2=0x7fabdc19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc101e50 0x7fabdc19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 -- 172.21.15.23:0/2738975911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 msgr2=0x7fabdc198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 0x7fabdc198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 -- 172.21.15.23:0/2738975911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4009d00 con 0x7fabdc1014c0 2023-11-17T00:07:47.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe219b700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 0x7fabdc198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe199a700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9a0 tx=0x7fabcc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc009790 con 0x7fabdc1014c0 2023-11-17T00:07:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc10bca0 con 0x7fabdc1014c0 2023-11-17T00:07:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabcc010430 con 0x7fabdc1014c0 2023-11-17T00:07:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc018680 con 0x7fabdc1014c0 2023-11-17T00:07:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.170+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc10bb10 con 0x7fabdc1014c0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.171+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc1a08b0 con 0x7fabdc1014c0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.173+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fabcc0105a0 con 0x7fabdc1014c0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.175+0000 7fabd37fe700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 0x7fabc80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.175+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9336+0+0 (secure 0 0 0) 0x7fabcc014070 con 0x7fabdc1014c0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.175+0000 7fabe219b700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 0x7fabc80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.176+0000 7fabe219b700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 0x7fabc80a3600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fabdc101320 tx=0x7fabd8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.179+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fabcc051730 con 0x7fabdc1014c0 2023-11-17T00:07:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-17T00:07:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T00:07:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-17T00:07:47.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6834/2058644251,v1:172.21.15.173:6835/2058644251] up:boot 2023-11-17T00:07:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: fsmap cephfs:0/3 7 up:standby, 3 failed 2023-11-17T00:07:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-17T00:07:47.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:46 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-17T00:07:47.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-17T00:07:47.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T00:07:47.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-17T00:07:47.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6834/2058644251,v1:172.21.15.173:6835/2058644251] up:boot 2023-11-17T00:07:47.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: fsmap cephfs:0/3 7 up:standby, 3 failed 2023-11-17T00:07:47.593 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-17T00:07:47.593 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:46 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-17T00:07:47.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.393+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fabdc1a0f50 con 0x7fabdc1014c0 2023-11-17T00:07:47.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-17T00:07:47.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T00:07:47.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/4110558949' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-17T00:07:47.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6834/2058644251,v1:172.21.15.173:6835/2058644251] up:boot 2023-11-17T00:07:47.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: fsmap cephfs:0/3 7 up:standby, 3 failed 2023-11-17T00:07:47.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-17T00:07:47.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:46 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-17T00:07:47.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.953+0000 7fabd37fe700 1 -- 172.21.15.23:0/2738975911 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 181+0+0 (secure 0 0 0) 0x7fabcc099cb0 con 0x7fabdc1014c0 2023-11-17T00:07:47.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 msgr2=0x7fabc80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:47.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 0x7fabc80a3600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fabdc101320 tx=0x7fabd8009260 comp rx=0 tx=0).stop 2023-11-17T00:07:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 msgr2=0x7fabdc1988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9a0 tx=0x7fabcc00ecb0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 shutdown_connections 2023-11-17T00:07:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7fabc80a1150 0x7fabc80a3600 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabdc101e50 0x7fabdc19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabdc1002d0 0x7fabdc198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 --2- 172.21.15.23:0/2738975911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabdc1014c0 0x7fabdc1988b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 >> 172.21.15.23:0/2738975911 conn(0x7fabdc0fb8b0 msgr2=0x7fabdc106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:47.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.957+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 shutdown_connections 2023-11-17T00:07:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:47.958+0000 7fabe43ff700 1 -- 172.21.15.23:0/2738975911 wait complete. 2023-11-17T00:07:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2023-11-17T00:07:47.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-17T00:07:48.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 -- 172.21.15.23:0/3803171780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 msgr2=0x7f9c5c103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:48.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 --2- 172.21.15.23:0/3803171780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c103180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009990 tx=0x7f9c58009ca0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 -- 172.21.15.23:0/3803171780 shutdown_connections 2023-11-17T00:07:48.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 --2- 172.21.15.23:0/3803171780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c105fe0 0x7f9c5c1083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 --2- 172.21.15.23:0/3803171780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c103180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 --2- 172.21.15.23:0/3803171780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c1036c0 0x7f9c5c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 -- 172.21.15.23:0/3803171780 >> 172.21.15.23:0/3803171780 conn(0x7f9c5c0fa7c0 msgr2=0x7f9c5c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:48.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 -- 172.21.15.23:0/3803171780 shutdown_connections 2023-11-17T00:07:48.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.196+0000 7f9c639ff700 1 -- 172.21.15.23:0/3803171780 wait complete. 2023-11-17T00:07:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.197+0000 7f9c639ff700 1 Processor -- start 2023-11-17T00:07:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.197+0000 7f9c639ff700 1 -- start start 2023-11-17T00:07:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c1036c0 0x7f9c5c1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c105fe0 0x7f9c5c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c19dc30 con 0x7f9c5c100da0 2023-11-17T00:07:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c19dd70 con 0x7f9c5c1036c0 2023-11-17T00:07:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c639ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c19df00 con 0x7f9c5c105fe0 2023-11-17T00:07:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c6179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c6179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48128/0 (socket says 172.21.15.23:48128) 2023-11-17T00:07:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c6179b700 1 -- 172.21.15.23:0/1837984430 learned_addr learned my addr 172.21.15.23:0/1837984430 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c61f9c700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c105fe0 0x7f9c5c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c60f9a700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c1036c0 0x7f9c5c1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c6179b700 1 -- 172.21.15.23:0/1837984430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c105fe0 msgr2=0x7f9c5c19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.198+0000 7f9c6179b700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c105fe0 0x7f9c5c19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c6179b700 1 -- 172.21.15.23:0/1837984430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c1036c0 msgr2=0x7f9c5c1965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c6179b700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c1036c0 0x7f9c5c1965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c6179b700 1 -- 172.21.15.23:0/1837984430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580095f0 con 0x7f9c5c100da0 2023-11-17T00:07:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c6179b700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b680 tx=0x7f9c58009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5801d070 con 0x7f9c5c100da0 2023-11-17T00:07:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c58021410 con 0x7f9c5c100da0 2023-11-17T00:07:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c10bc70 con 0x7f9c5c100da0 2023-11-17T00:07:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58017650 con 0x7f9c5c100da0 2023-11-17T00:07:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.199+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c10bae0 con 0x7f9c5c100da0 2023-11-17T00:07:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.200+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c0fc330 con 0x7f9c5c100da0 2023-11-17T00:07:48.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.202+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9c58021580 con 0x7f9c5c100da0 2023-11-17T00:07:48.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.203+0000 7f9c527fc700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 0x7f9c480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.204+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8811+0+0 (secure 0 0 0) 0x7f9c580cd9f0 con 0x7f9c5c100da0 2023-11-17T00:07:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.204+0000 7f9c60f9a700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 0x7f9c480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.204+0000 7f9c60f9a700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 0x7f9c480a3600 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009c60 tx=0x7f9c4c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.207+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9c58096f30 con 0x7f9c5c100da0 2023-11-17T00:07:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-17T00:07:48.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T00:07:48.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: Cluster is now healthy 2023-11-17T00:07:48.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6832/3001298117,v1:172.21.15.173:6833/3001298117] up:boot 2023-11-17T00:07:48.270 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.173:6838/1288557849,v1:172.21.15.173:6839/1288557849] up:boot 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6834/2489715234,v1:172.21.15.156:6835/2489715234] up:boot 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6836/4239202046,v1:172.21.15.156:6837/4239202046] up:boot 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: mds.? [v2:172.21.15.156:6838/1752279184,v1:172.21.15.156:6839/1752279184] up:boot 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: fsmap 12 up:standby 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-17T00:07:48.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-17T00:07:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-17T00:07:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-17T00:07:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:07:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:47 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-17T00:07:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T00:07:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: Cluster is now healthy 2023-11-17T00:07:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6832/3001298117,v1:172.21.15.173:6833/3001298117] up:boot 2023-11-17T00:07:48.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.173:6838/1288557849,v1:172.21.15.173:6839/1288557849] up:boot 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6834/2489715234,v1:172.21.15.156:6835/2489715234] up:boot 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6836/4239202046,v1:172.21.15.156:6837/4239202046] up:boot 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: mds.? [v2:172.21.15.156:6838/1752279184,v1:172.21.15.156:6839/1752279184] up:boot 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: fsmap 12 up:standby 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-17T00:07:48.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:47 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: Cluster is now healthy 2023-11-17T00:07:48.360 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6832/3001298117,v1:172.21.15.173:6833/3001298117] up:boot 2023-11-17T00:07:48.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.173:6838/1288557849,v1:172.21.15.173:6839/1288557849] up:boot 2023-11-17T00:07:48.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6834/2489715234,v1:172.21.15.156:6835/2489715234] up:boot 2023-11-17T00:07:48.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6836/4239202046,v1:172.21.15.156:6837/4239202046] up:boot 2023-11-17T00:07:48.361 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: mds.? [v2:172.21.15.156:6838/1752279184,v1:172.21.15.156:6839/1752279184] up:boot 2023-11-17T00:07:48.364 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/3985628' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-17T00:07:48.364 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: fsmap 12 up:standby 2023-11-17T00:07:48.364 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-17T00:07:48.364 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-17T00:07:48.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-17T00:07:48.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-17T00:07:48.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='mgr.14150 172.21.15.23:0/911245340' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-17T00:07:48.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: pgmap v572: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2023-11-17T00:07:48.365 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:47 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:48.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.416+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c5c02cfe0 con 0x7f9c5c100da0 2023-11-17T00:07:48.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.948+0000 7f9c527fc700 1 -- 172.21.15.23:0/1837984430 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9c5809a550 con 0x7f9c5c100da0 2023-11-17T00:07:48.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.953+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 msgr2=0x7f9c480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:48.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.953+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 0x7f9c480a3600 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009c60 tx=0x7f9c4c009260 comp rx=0 tx=0).stop 2023-11-17T00:07:48.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 msgr2=0x7f9c5c1960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:48.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b680 tx=0x7f9c58009580 comp rx=0 tx=0).stop 2023-11-17T00:07:48.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 shutdown_connections 2023-11-17T00:07:48.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f9c480a1150 0x7f9c480a3600 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c5c105fe0 0x7f9c5c19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c5c100da0 0x7f9c5c1960b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 --2- 172.21.15.23:0/1837984430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c5c1036c0 0x7f9c5c1965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:48.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.954+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 >> 172.21.15.23:0/1837984430 conn(0x7f9c5c0fa7c0 msgr2=0x7f9c5c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:48.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.955+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 shutdown_connections 2023-11-17T00:07:48.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:48.955+0000 7f9c639ff700 1 -- 172.21.15.23:0/1837984430 wait complete. 2023-11-17T00:07:48.961 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2023-11-17T00:07:48.994 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-17T00:07:49.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 -- 172.21.15.23:0/714567153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 msgr2=0x7f0fcc108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/714567153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc108710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400b0a0 tx=0x7f0fc400b3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 -- 172.21.15.23:0/714567153 shutdown_connections 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/714567153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc108710 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/714567153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1002d0 0x7f0fcc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/714567153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 -- 172.21.15.23:0/714567153 >> 172.21.15.23:0/714567153 conn(0x7f0fcc0fb8b0 msgr2=0x7f0fcc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:49.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 -- 172.21.15.23:0/714567153 shutdown_connections 2023-11-17T00:07:49.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.209+0000 7f0fd27ff700 1 -- 172.21.15.23:0/714567153 wait complete. 2023-11-17T00:07:49.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.210+0000 7f0fd27ff700 1 Processor -- start 2023-11-17T00:07:49.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.210+0000 7f0fd27ff700 1 -- start start 2023-11-17T00:07:49.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1002d0 0x7f0fcc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:49.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:49.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc19fc60 con 0x7f0fcc1014c0 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc19fda0 con 0x7f0fcc1002d0 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd27ff700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc19ff30 con 0x7f0fcc101e50 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fcbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1002d0 0x7f0fcc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.23:33374/0 (socket says 172.21.15.23:33374) 2023-11-17T00:07:49.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 -- 172.21.15.23:0/1300809538 learned_addr learned my addr 172.21.15.23:0/1300809538 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 -- 172.21.15.23:0/1300809538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1002d0 msgr2=0x7f0fcc1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1002d0 0x7f0fcc1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 -- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 msgr2=0x7f0fcc198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fd0d9c700 1 -- 172.21.15.23:0/1300809538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4009d00 con 0x7f0fcc101e50 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.211+0000 7f0fcb7fe700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-17T00:07:49.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fd0d9c700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009850 tx=0x7f0fc400bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400e040 con 0x7f0fcc101e50 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc10bbe0 con 0x7f0fcc101e50 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fc4003e10 con 0x7f0fcc101e50 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc401b6c0 con 0x7f0fcc101e50 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.212+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc10ba50 con 0x7f0fcc101e50 2023-11-17T00:07:49.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.213+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc067b00 con 0x7f0fcc101e50 2023-11-17T00:07:49.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.215+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0fc401b820 con 0x7f0fcc101e50 2023-11-17T00:07:49.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.217+0000 7f0fc97fa700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 0x7f0fb40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-17T00:07:49.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.217+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8252+0+0 (secure 0 0 0) 0x7f0fc40ce500 con 0x7f0fcc101e50 2023-11-17T00:07:49.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.217+0000 7f0fcbfff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 0x7f0fb40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-17T00:07:49.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.217+0000 7f0fcbfff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 0x7f0fb40a3600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc101320 tx=0x7f0fbc00a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T00:07:49.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.219+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0fc4097c70 con 0x7f0fcc101e50 2023-11-17T00:07:49.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:48 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:49.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:48 smithi023 ceph-mon[97824]: osdmap e95: 12 total, 12 up, 12 in 2023-11-17T00:07:49.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:48 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:49.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:48 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:49.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:48 smithi173 ceph-mon[131534]: osdmap e95: 12 total, 12 up, 12 in 2023-11-17T00:07:49.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:48 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:49.361 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:48 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/2738975911' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:49.361 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:48 smithi156 ceph-mon[131105]: osdmap e95: 12 total, 12 up, 12 in 2023-11-17T00:07:49.362 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:48 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:49.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.439+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0fcc1a0aa0 con 0x7f0fcc101e50 2023-11-17T00:07:49.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.959+0000 7f0fc97fa700 1 -- 172.21.15.23:0/1300809538 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) v1 ==== 178+0+0 (secure 0 0 0) 0x7f0fcc1a0aa0 con 0x7f0fcc101e50 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 msgr2=0x7f0fb40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 0x7f0fb40a3600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc101320 tx=0x7f0fbc00a3b0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 msgr2=0x7f0fcc19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009850 tx=0x7f0fc400bce0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 shutdown_connections 2023-11-17T00:07:49.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:6800/499313593,v1:172.21.15.23:6801/499313593] conn(0x7f0fb40a1150 0x7f0fb40a3600 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fcc101e50 0x7f0fcc19f690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fcc1002d0 0x7f0fcc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 --2- 172.21.15.23:0/1300809538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc1014c0 0x7f0fcc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.963+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 >> 172.21.15.23:0/1300809538 conn(0x7f0fcc0fb8b0 msgr2=0x7f0fcc0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.964+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 shutdown_connections 2023-11-17T00:07:49.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-11-17T00:07:49.964+0000 7f0fd27ff700 1 -- 172.21.15.23:0/1300809538 wait complete. 2023-11-17T00:07:49.967 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' removed 2023-11-17T00:07:49.995 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:07:50.067 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:07:50.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:07:50.138 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-17T00:07:50.138 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T00:07:50.184 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T00:07:50.202 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T00:07:50.221 INFO:tasks.cephadm:Stopping all daemons... 2023-11-17T00:07:50.221 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-11-17T00:07:50.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a 2023-11-17T00:07:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:49 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:49 smithi023 ceph-mon[97824]: osdmap e96: 12 total, 12 up, 12 in 2023-11-17T00:07:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:49 smithi023 ceph-mon[97824]: pgmap v575: 33 pgs: 33 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-17T00:07:50.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:49 smithi023 ceph-mon[97824]: from='client.? 172.21.15.23:0/1300809538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.269 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:49 smithi023 ceph-mon[97824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:49 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:49 smithi173 ceph-mon[131534]: osdmap e96: 12 total, 12 up, 12 in 2023-11-17T00:07:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:49 smithi173 ceph-mon[131534]: pgmap v575: 33 pgs: 33 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-17T00:07:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:49 smithi173 ceph-mon[131534]: from='client.? 172.21.15.23:0/1300809538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:49 smithi173 ceph-mon[131534]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:49 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1837984430' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-17T00:07:50.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:49 smithi156 ceph-mon[131105]: osdmap e96: 12 total, 12 up, 12 in 2023-11-17T00:07:50.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:49 smithi156 ceph-mon[131105]: pgmap v575: 33 pgs: 33 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-17T00:07:50.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:49 smithi156 ceph-mon[131105]: from='client.? 172.21.15.23:0/1300809538' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:49 smithi156 ceph-mon[131105]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-17T00:07:50.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:50 smithi023 systemd[1]: Stopping Ceph mon.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:51.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a[97800]: 2023-11-17T00:07:50.807+0000 7f6f2c184700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:07:51.123 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a[97800]: 2023-11-17T00:07:50.807+0000 7f6f2c184700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-11-17T00:07:51.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Nov 17 00:07:51 smithi023 bash[253852]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-a 2023-11-17T00:07:51.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.a.service' 2023-11-17T00:07:51.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:51.828 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-11-17T00:07:51.828 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-11-17T00:07:51.828 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.b 2023-11-17T00:07:52.203 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:51 smithi156 systemd[1]: Stopping Ceph mon.b for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:52.504 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-b[131081]: 2023-11-17T00:07:52.201+0000 7f90c512e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:07:52.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-b[131081]: 2023-11-17T00:07:52.201+0000 7f90c512e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-11-17T00:07:53.034 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.b.service' 2023-11-17T00:07:53.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:52 smithi156 bash[160527]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-b 2023-11-17T00:07:53.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:53 smithi156 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.b.service: Succeeded. 2023-11-17T00:07:53.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Nov 17 00:07:53 smithi156 systemd[1]: Stopped Ceph mon.b for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:07:53.088 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:53.089 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-11-17T00:07:53.089 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-11-17T00:07:53.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.c 2023-11-17T00:07:53.446 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:53 smithi173 systemd[1]: Stopping Ceph mon.c for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:53.734 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-c[131495]: 2023-11-17T00:07:53.444+0000 7fb2e2e49700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:07:53.734 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-c[131495]: 2023-11-17T00:07:53.444+0000 7fb2e2e49700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-11-17T00:07:54.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:Nov 17 00:07:53 smithi173 bash[162009]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mon-c 2023-11-17T00:07:54.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mon.c.service' 2023-11-17T00:07:54.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:54.278 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-11-17T00:07:54.278 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-11-17T00:07:54.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-17T00:07:54.768 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 17 00:07:54 smithi023 systemd[1]: Stopping Ceph mgr.x for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:55.086 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 17 00:07:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-17T00:07:54.790+0000 7f3b4bd74700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:07:55.086 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 17 00:07:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x[98072]: 2023-11-17T00:07:54.790+0000 7f3b4bd74700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-17T00:07:55.518 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Nov 17 00:07:55 smithi023 bash[254001]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-x 2023-11-17T00:07:55.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.x.service' 2023-11-17T00:07:55.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:55.678 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-11-17T00:07:55.678 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-11-17T00:07:55.678 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.y 2023-11-17T00:07:56.107 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 17 00:07:55 smithi156 systemd[1]: Stopping Ceph mgr.y for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:56.561 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 17 00:07:56 smithi156 bash[160678]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-y 2023-11-17T00:07:56.562 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Nov 17 00:07:56 smithi156 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-11-17T00:07:56.569 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.y.service' 2023-11-17T00:07:56.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:56.610 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-11-17T00:07:56.611 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-11-17T00:07:56.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.z 2023-11-17T00:07:57.042 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 17 00:07:56 smithi173 systemd[1]: Stopping Ceph mgr.z for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:57.323 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Nov 17 00:07:57 smithi173 bash[162162]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mgr-z 2023-11-17T00:07:57.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mgr.z.service' 2023-11-17T00:07:57.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:07:57.556 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-11-17T00:07:57.556 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-11-17T00:07:57.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.0 2023-11-17T00:07:58.018 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:07:57 smithi023 systemd[1]: Stopping Ceph osd.0 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:07:58.518 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:07:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0[110064]: 2023-11-17T00:07:58.072+0000 7f8373860700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:07:58.519 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:07:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0[110064]: 2023-11-17T00:07:58.072+0000 7f8373860700 -1 osd.0 97 *** Got signal Terminated *** 2023-11-17T00:07:58.519 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:07:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0[110064]: 2023-11-17T00:07:58.072+0000 7f8373860700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:04.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 17 00:08:03 smithi023 bash[254151]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-0 2023-11-17T00:08:07.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.0.service' 2023-11-17T00:08:07.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:07.427 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-11-17T00:08:07.427 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-11-17T00:08:07.427 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.1 2023-11-17T00:08:07.829 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:08:07 smithi156 systemd[1]: Stopping Ceph osd.1 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:08.107 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:08:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-17T00:08:07.827+0000 7fddcc16c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:08.108 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:08:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-17T00:08:07.827+0000 7fddcc16c700 -1 osd.1 97 *** Got signal Terminated *** 2023-11-17T00:08:08.108 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:08:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1[136190]: 2023-11-17T00:08:07.827+0000 7fddcc16c700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:13.857 INFO:journalctl@ceph.osd.1.smithi156.stdout:Nov 17 00:08:13 smithi156 bash[160835]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-1 2023-11-17T00:08:14.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.1.service' 2023-11-17T00:08:14.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:14.967 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-11-17T00:08:14.968 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-11-17T00:08:14.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.2 2023-11-17T00:08:15.327 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:08:15 smithi173 systemd[1]: Stopping Ceph osd.2 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:15.607 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:08:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-17T00:08:15.324+0000 7f3cc8721700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:15.607 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:08:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-17T00:08:15.324+0000 7f3cc8721700 -1 osd.2 97 *** Got signal Terminated *** 2023-11-17T00:08:15.607 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:08:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2[137399]: 2023-11-17T00:08:15.324+0000 7f3cc8721700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:21.103 INFO:journalctl@ceph.osd.2.smithi173.stdout:Nov 17 00:08:21 smithi173 bash[162307]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-2 2023-11-17T00:08:22.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.2.service' 2023-11-17T00:08:22.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:22.410 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-11-17T00:08:22.410 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-11-17T00:08:22.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.3 2023-11-17T00:08:22.768 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:22 smithi023 systemd[1]: Stopping Ceph osd.3 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:23.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:22.936+0000 7fdde6a8c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:23.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:22.936+0000 7fdde6a8c700 -1 osd.3 97 *** Got signal Terminated *** 2023-11-17T00:08:23.268 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:22 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:22.936+0000 7fdde6a8c700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:25.857 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:25.365+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.632 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:25.926+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.633 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:26.378+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.633 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:25 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:25.922+0000 7fdde107e700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:07:59.674519+0000 front 2023-11-17T00:07:59.674829+0000 (oldest deadline 2023-11-17T00:08:25.574667+0000) 2023-11-17T00:08:31.633 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:26 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:26.903+0000 7fdde107e700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:07:59.674519+0000 front 2023-11-17T00:07:59.674829+0000 (oldest deadline 2023-11-17T00:08:25.574667+0000) 2023-11-17T00:08:31.633 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:27 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3[115508]: 2023-11-17T00:08:27.879+0000 7fdde107e700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:07:59.674519+0000 front 2023-11-17T00:07:59.674829+0000 (oldest deadline 2023-11-17T00:08:25.574667+0000) 2023-11-17T00:08:31.634 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:28 smithi023 bash[254442]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-3 2023-11-17T00:08:31.644 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:26.955+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.645 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:28 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:28.862+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:31.645 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:29 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:29.876+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:31.645 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:25 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:25.583+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.645 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:26 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:26.540+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.646 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:27 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:27.573+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.646 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:28 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:28.542+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.646 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:29 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:29.502+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.646 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:30 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:30.499+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:31.646 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:27.356+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.647 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:27.926+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.647 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:28.308+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.3.service' 2023-11-17T00:08:31.648 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:30 smithi023 systemd[1]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.3.service: Succeeded. 2023-11-17T00:08:31.648 INFO:journalctl@ceph.osd.3.smithi023.stdout:Nov 17 00:08:30 smithi023 systemd[1]: Stopped Ceph osd.3 for d04064ea-84da-11ee-8dbe-212e2dc638e7. 2023-11-17T00:08:31.649 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:30 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:30.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:31.649 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:28.958+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.650 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:29.351+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.650 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:29.932+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.650 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:30.323+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.651 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:30.959+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:31.651 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:31 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:31.300+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:31.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:31.748 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-11-17T00:08:31.749 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-11-17T00:08:31.749 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.4 2023-11-17T00:08:31.857 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:31 smithi156 systemd[1]: Stopping Ceph osd.4 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:31.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:31 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:31.543+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:32.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:31 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:31.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:32.342 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:31 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:31.993+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:32.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:32 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-17T00:08:32.114+0000 7fb031a9b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:32.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:32 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-17T00:08:32.114+0000 7fb031a9b700 -1 osd.4 97 *** Got signal Terminated *** 2023-11-17T00:08:32.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:32 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-17T00:08:32.114+0000 7fb031a9b700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:32.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:32 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:32.341+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:32.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:32 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:32.565+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:33.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:32 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:32.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:33.317 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:33 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:33.006+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:33.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:33 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:33.315+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:33.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:33 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:33.558+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:34.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:33 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:33.861+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:34.286 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:34 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:34.005+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:34.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:34 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:34.284+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:34.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:34 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:34.284+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:34.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:34 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:34.572+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:35.268 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:34 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:34.968+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:35.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:34 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:34.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:35.287 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:34 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:34.975+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:35.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:35 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:35.286+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:35.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:35 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:35.286+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:35.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:35 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:35.587+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:36.250 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:35 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:35.954+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:36.251 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:35 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:35.954+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:36.268 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:35.941+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:36.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:35.874+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:36.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:35 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:35.874+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:36.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:36.249+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:36.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:36.249+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:36.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:36 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:36.553+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:37.247 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:36.906+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:37.247 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:36 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:36.906+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:37.269 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:36.985+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:37.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:36.858+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:37.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:36.858+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:37.357 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:36 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4[141694]: 2023-11-17T00:08:36.949+0000 7fb02c08d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.506358+0000 front 2023-11-17T00:08:12.506324+0000 (oldest deadline 2023-11-17T00:08:36.606461+0000) 2023-11-17T00:08:37.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:37 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:37.246+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:37.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:37 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:37.246+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:37.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:37 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:37.563+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:37.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:37 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:37.563+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:38.268 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:38 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:38.009+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:38.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:37 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:37.809+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:38.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:37 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:37.809+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:38.288 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:37 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:37.889+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:38.289 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:37 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:37.889+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:38.358 INFO:journalctl@ceph.osd.4.smithi156.stdout:Nov 17 00:08:37 smithi156 bash[161104]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-4 2023-11-17T00:08:38.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:37 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:37.878+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:38.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:38 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:38.327+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:38.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:38 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:38.286+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:38.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:38 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:38.286+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:38.816 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:38 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:38.525+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:38.816 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:38 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:38.525+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:39.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:38 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:38.828+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:39.156 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.4.service' 2023-11-17T00:08:39.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:39.196 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-11-17T00:08:39.196 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-11-17T00:08:39.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.5 2023-11-17T00:08:39.227 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:38 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:38.938+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:39.227 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:38 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:38.938+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:39.268 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:38 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:38.988+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:39.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:38 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:38.858+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:39.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:38 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:38.858+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:39.554 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:39.280+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:39.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 systemd[1]: Stopping Ceph osd.5 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:39.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:39.317+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:39.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:39.317+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:39.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:39.553+0000 7f53cf6a9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:39.857 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:39.553+0000 7f53cf6a9700 -1 osd.5 97 *** Got signal Terminated *** 2023-11-17T00:08:39.857 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:39.553+0000 7f53cf6a9700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:39.857 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:39 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:39.790+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:39.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:39 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:39.554+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:39.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:39 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:39.554+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:40.268 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:40 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:40.033+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:40.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:39 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:39.832+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:40.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:39 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:39.832+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:40.303 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:39.949+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:40.303 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:39 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:39.949+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:40.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:40 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:40.302+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:40.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:40 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:40.314+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:40.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:40 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:40.314+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:40.811 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:40 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:40.516+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:40.811 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:40 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:40.516+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:41.068 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:40 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:40.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:41.068 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:40 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:40.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:41.109 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:40 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:40.809+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:41.299 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:40 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:40.954+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:41.299 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:40 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:40.954+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:41.519 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:41.067+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:41.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:41 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:41.298+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:41.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:41 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:41.353+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:41.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:41 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:41.353+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:41.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:41 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:41.552+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:41.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:41 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:41.552+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:42.066 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:41.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:42.066 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:41.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:42.066 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:41.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:42.280 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:41 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:41.930+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:42.280 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:41 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:41.930+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:42.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:41 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:41.859+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:42.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:42.065+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:42.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:42.279+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:42.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:42.328+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:42.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:42.328+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:42.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:42.328+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:18.338870+0000 front 2023-11-17T00:08:18.338898+0000 (oldest deadline 2023-11-17T00:08:41.839005+0000) 2023-11-17T00:08:42.813 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:42 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:42.530+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:42.814 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:42 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:42.530+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:43.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:42.756+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:43.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:42.756+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:43.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:42 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:42.756+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:43.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:42 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:42.812+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:43.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:42 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:42.812+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:43.259 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:42.912+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:43.259 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:42.912+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:43.259 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:42 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:42.912+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:43.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:43.091+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:43.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:43.258+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:43.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:43.351+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:43.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:43.351+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:43.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:43.351+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:18.338870+0000 front 2023-11-17T00:08:18.338898+0000 (oldest deadline 2023-11-17T00:08:41.839005+0000) 2023-11-17T00:08:43.857 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:43 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:43.763+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:43.857 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:43 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:43.763+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:43.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:43 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:43.547+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:43.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:43 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:43.547+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:44.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:43.717+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:44.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:43.717+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:44.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:43 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:43.717+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:44.283 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:43.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:44.413 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:43.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:44.413 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:43 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:43.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:44.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:44.139+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:44.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:44.279+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:44.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:44.324+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:00.434498+0000 front 2023-11-17T00:08:00.434376+0000 (oldest deadline 2023-11-17T00:08:24.534513+0000) 2023-11-17T00:08:44.607 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:44.324+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.436927+0000 front 2023-11-17T00:08:11.436983+0000 (oldest deadline 2023-11-17T00:08:33.737050+0000) 2023-11-17T00:08:44.608 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5[142796]: 2023-11-17T00:08:44.324+0000 7f53c7c97700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:18.338870+0000 front 2023-11-17T00:08:18.338898+0000 (oldest deadline 2023-11-17T00:08:41.839005+0000) 2023-11-17T00:08:44.857 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:44 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:44.787+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:44.857 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:44 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:44.787+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:44.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:44 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:44.569+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:44.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:44 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:44.569+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:45.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:44.734+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:45.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:44.734+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:45.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:44.734+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:45.233 INFO:journalctl@ceph.osd.5.smithi173.stdout:Nov 17 00:08:45 smithi173 bash[162571]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-5 2023-11-17T00:08:45.233 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:44.963+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:45.234 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:44.963+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:45.234 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:44 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:44.963+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:45.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:45.101+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:45.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:45.101+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:45.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:45.294+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:45.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:45.294+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:45.834 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:45.547+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:45.834 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:45.547+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:46.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:45.735+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:46.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:45.735+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:46.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:45 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:45.735+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:46.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:45.832+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:46.108 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:45 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:45.832+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:46.108 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:45.957+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:46.108 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:45.957+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:46.108 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:45 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:45.958+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:46.491 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:46.263+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:46.491 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:46.263+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:46.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.5.service' 2023-11-17T00:08:46.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:46.127+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:46.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:46.127+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:46.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:46.579 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-11-17T00:08:46.579 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-11-17T00:08:46.579 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.6 2023-11-17T00:08:46.858 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:46 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:46.794+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:46.858 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:46 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:46.794+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:46.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:46 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:46.558+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:46.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:46 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:46.558+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:46.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:46 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:46.558+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:47.018 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:46 smithi023 systemd[1]: Stopping Ceph osd.6 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:47.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:46.726+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:47.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:46.726+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:47.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:46 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:46.726+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:47.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:46.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:47.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:46.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:47.278 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:46 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:46.915+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:47.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:47.100+0000 7fdfbe8f0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:47.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:47.100+0000 7fdfbe8f0700 -1 osd.6 97 *** Got signal Terminated *** 2023-11-17T00:08:47.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:47.100+0000 7fdfbe8f0700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:47.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:47.124+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:47.519 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:47.124+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:47.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:47.277+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:47.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:47.277+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:47.839 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:47.551+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:47.839 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:47.551+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:47.840 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:47.551+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:48.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:47.838+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:48.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:47 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:47.838+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:48.171 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:47.774+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:48.171 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:47.774+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:48.171 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:47 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:47.774+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:48.299 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:47.959+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:48.299 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:47.959+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:48.299 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:47 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:47.959+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:48.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:48.170+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:48.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:48.170+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:48.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:48.298+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:48.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:48.298+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:48.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:48.528+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:48.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:48.528+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:48.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:48.528+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:49.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:48.763+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:49.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:48.763+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:49.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:48 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:48.763+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:49.329 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:48.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:49.330 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:48.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:49.330 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:48 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:48.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:49.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:48.873+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:49.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:48 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:48.873+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:49.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:49.139+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:49.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:49.139+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:49.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:49.328+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:49.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:49.328+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:49.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:49.328+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:49.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:49.491+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:49.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:49.491+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:49.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:49.491+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:50.119 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:49.776+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:50.119 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:49.776+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:50.120 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:49.776+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:50.120 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:49 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:49.776+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:50.356 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:49.994+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:50.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:49.994+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:50.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:49 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:49.994+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:50.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:49.860+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:50.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:49 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:49.860+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:50.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:50.118+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:50.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:50.118+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:50.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:50.355+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:50.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:50.355+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:50.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:50 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:50.355+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:50.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:50.530+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:50.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:50.530+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:50.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:50.530+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:51.090 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:50.785+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:51.091 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:50.785+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:51.091 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:50.785+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:51.091 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:50 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:50.785+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.011+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.011+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:51.342 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.011+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:51.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:50.900+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:51.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:50.900+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:51.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:51.090+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.008093+0000 front 2023-11-17T00:08:12.008003+0000 (oldest deadline 2023-11-17T00:08:34.908128+0000) 2023-11-17T00:08:51.518 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6[120933]: 2023-11-17T00:08:51.090+0000 7fdfba6e5700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.510166+0000 front 2023-11-17T00:08:19.510261+0000 (oldest deadline 2023-11-17T00:08:44.210326+0000) 2023-11-17T00:08:51.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:51.340+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:51.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:51.340+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:51.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:51.340+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:51.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:51.484+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:51.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:51.484+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:51.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:51.485+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:51.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:51.485+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:52.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:51.829+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:52.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:51.829+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:52.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:51.829+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:52.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:51 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:51.829+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:52.317 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.982+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:52.317 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.982+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:52.317 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:51 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:51.982+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:52.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:51.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:52.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:51 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:51.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:52.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:52.315+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:52.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:52.315+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:52.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:52.315+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:52.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:52.497+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:52.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:52.497+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:52.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:52.497+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:52.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:52.497+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:52.898 INFO:journalctl@ceph.osd.6.smithi023.stdout:Nov 17 00:08:52 smithi023 bash[254713]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-6 2023-11-17T00:08:52.898 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:52 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:52.789+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:52.898 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:52 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:52.789+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:52.898 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:52 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:52.789+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:52.899 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:52 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:52.789+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:53.288 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:52.976+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:53.288 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:52.976+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:53.288 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:52 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:52.976+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:53.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:52.947+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:53.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:52.948+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:53.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:52 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:52.948+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:53.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:53.286+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:53.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:53.286+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:53.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:53.286+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:53.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:53.496+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:53.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:53.496+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:53.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:53.496+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:53.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:53.496+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:54.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:53 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:53.803+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:54.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:53 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:53.803+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:54.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:53 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:53.803+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:54.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:53 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:53.803+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:54.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:53.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:54.286 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:53.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:54.286 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:53 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:53.995+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:54.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:53.942+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:54.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:53.942+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:54.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:53 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:53.942+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:54.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:54.283+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:54.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:54.283+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:54.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:54.283+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:54.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.6.service' 2023-11-17T00:08:54.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:08:54.854 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-11-17T00:08:54.855 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-11-17T00:08:54.855 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.7 2023-11-17T00:08:54.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:54.447+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:54.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:54.447+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:54.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:54.447+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:54.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:54.447+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:55.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:54.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:55.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:54.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:55.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:54.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:55.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:54 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:54.800+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:55.220 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:54 smithi156 systemd[1]: Stopping Ceph osd.7 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:54.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:54.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:54.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.219+0000 7f78cc96b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.219+0000 7f78cc96b700 -1 osd.7 97 *** Got signal Terminated *** 2023-11-17T00:08:55.221 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.219+0000 7f78cc96b700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:08:55.294 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:54.958+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:55.294 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:54.958+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:55.295 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:54 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:54.958+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:55.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:55.293+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:55.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:55.293+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:55.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:55.293+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:55.607 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:55.417+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:55.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:55.417+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:55.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:55.417+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:55.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:55.417+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:56.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:55.987+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:56.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:55.987+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:56.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:55 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:55.987+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:56.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:55.839+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:56.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:55.839+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:56.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:55.839+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:56.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:55 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:55.839+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:56.359 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.875+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:56.359 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.875+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:56.360 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:55 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:55.875+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:56.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:56 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:56.253+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:56.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:56 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:56.253+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:56.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:56 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:56.253+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:56.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:56.421+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:56.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:56.421+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:56.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:56.421+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:56.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:56.421+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:57.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:56.824+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:57.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:56.824+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:57.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:56.824+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:57.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:56 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:56.824+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:57.358 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:57.262+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:57.361 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:57.262+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:57.361 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:57.262+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:57.362 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:57.034+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:57.363 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:57.034+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:57.363 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:57 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:57.034+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:57.363 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:56.910+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:57.363 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:56.910+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:57.363 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:56 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:56.910+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:57.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:57.403+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:57.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:57.403+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:57.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:57.403+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:57.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:57.403+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:58.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:57.777+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:58.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:57.777+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:58.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:57.777+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:58.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:57 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:57.777+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:58.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:57.882+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:58.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:57.882+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:58.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:57 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:57.882+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:58.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:58.066+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:58.066+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:58.066+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:58.287+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:58.287+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:58.287+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:58.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:58 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:58.287+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:08:58.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:58.391+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:58.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:58.391+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:58.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:58.391+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:58.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:58.391+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:08:59.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:58.821+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:08:59.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:58.821+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:08:59.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:58.821+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:08:59.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:58 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:58.821+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:08:59.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:58.870+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:08:59.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:58.870+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:08:59.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:58.870+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:08:59.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:58 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:58.870+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:36.838762+0000 front 2023-11-17T00:08:36.838781+0000 (oldest deadline 2023-11-17T00:08:58.538802+0000) 2023-11-17T00:08:59.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:59.297+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:08:59.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:59.297+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:08:59.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:59.297+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:08:59.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:08:59.297+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:08:59.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:59.113+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:08:59.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:59.113+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:08:59.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:59.113+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:08:59.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:08:59 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:08:59.113+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:08:59.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:59.418+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:08:59.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:59.418+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:08:59.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:59.418+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:08:59.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:08:59.418+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:00.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:59 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:59.871+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:00.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:59 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:59.871+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:00.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:59 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:59.871+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:00.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:08:59 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:08:59.871+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:00.357 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:59.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.235101+0000 front 2023-11-17T00:08:12.235139+0000 (oldest deadline 2023-11-17T00:08:36.935262+0000) 2023-11-17T00:09:00.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:59.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:16.935961+0000 front 2023-11-17T00:08:16.935832+0000 (oldest deadline 2023-11-17T00:08:42.235924+0000) 2023-11-17T00:09:00.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:59.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.337303+0000 front 2023-11-17T00:08:26.337372+0000 (oldest deadline 2023-11-17T00:08:52.237445+0000) 2023-11-17T00:09:00.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:08:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7[147097]: 2023-11-17T00:08:59.909+0000 7f78c575a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:36.838762+0000 front 2023-11-17T00:08:36.838781+0000 (oldest deadline 2023-11-17T00:08:58.538802+0000) 2023-11-17T00:09:00.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:00.254+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:00.254+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:00.254+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:00.254+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:00.151+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:00.151+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:00.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:00.151+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:00.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:00 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:00.151+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:00.830 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:00 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:00.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:00.830 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:00 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:00.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:00.831 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:00 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:00.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:00.831 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:00 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:00.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:00.831 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:00 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:00.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:01.107 INFO:journalctl@ceph.osd.7.smithi156.stdout:Nov 17 00:09:00 smithi156 bash[161369]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-7 2023-11-17T00:09:01.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:00.822+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:01.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:00.822+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:01.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:00.822+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:01.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:00 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:00.822+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:01.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:01.238+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:01.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:01.238+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:01.238+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:01.238+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:01.115+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:01.115+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:01.115+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:01.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:01 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:01.115+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:01.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:01 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:01.376+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:01.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:01 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:01.376+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:01.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:01 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:01.376+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:01.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:01 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:01.376+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:01.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:01 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:01.376+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:02.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:01.852+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:02.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:01.852+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:02.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:01.852+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:02.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:01.852+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:02.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:01 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:01.852+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:02.374 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.7.service' 2023-11-17T00:09:02.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:02.417 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-11-17T00:09:02.417 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-11-17T00:09:02.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.8 2023-11-17T00:09:02.449 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.275+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:02.449 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.275+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:02.449 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.275+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:02.449 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.275+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:02.449 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:02.118+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:02.450 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:02.118+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:02.450 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:02.118+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:02.450 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:02.118+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:02.777 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 systemd[1]: Stopping Ceph osd.8 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:02.777 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.774+0000 7fd2ca525700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:02.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:02 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:02.405+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:02.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:02 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:02.405+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:02.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:02 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:02.405+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:02.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:02 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:02.405+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:02.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:02 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:02.405+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:03.107 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.774+0000 7fd2ca525700 -1 osd.8 97 *** Got signal Terminated *** 2023-11-17T00:09:03.107 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:02 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:02.774+0000 7fd2ca525700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:09:03.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:02.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:03.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:02.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:03.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:02.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:03.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:02.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:03.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:02 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:02.840+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:03.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:03.269+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:03.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:03.269+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:03.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:03.269+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:03.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:03.269+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:03.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:03.161+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:03.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:03.161+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:03.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:03.161+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:03.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:03 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:03.161+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:03.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:03.433+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:03.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:03.433+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:03.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:03.433+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:03.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:03.433+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:03.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:03.433+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:04.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:03.875+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:04.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:03.875+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:04.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:03.875+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:04.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:03.875+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:04.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:03 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:03.875+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:04.607 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:04.244+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:04.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:04.244+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:04.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:04.244+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:04.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:04.244+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:04.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:04.142+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:04.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:04.142+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:04.142+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:04 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:04.142+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:04.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:04 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:04.465+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:04.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:04 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:04.465+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:04.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:04 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:04.465+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:04.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:04 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:04.465+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:04.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:04 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:04.465+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:05.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:04.847+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:05.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:04.847+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:05.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:04.847+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:05.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:04.847+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:05.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:04 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:04.847+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:05.357 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:05.223+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:05.357 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:05.223+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:05.358 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:05.223+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:05.358 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:05.223+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:05.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:05.103+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:05.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:05.103+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:05.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:05.103+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:05.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:05.103+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:05.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:05 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:05.103+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:05.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:05 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:05.454+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:05.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:05 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:05.454+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:05.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:05 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:05.454+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:05.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:05 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:05.454+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:05.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:05 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:05.454+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:06.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:05 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:05.823+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:06.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:05 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:05.823+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:06.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:05 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:05.823+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:06.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:05 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:05.823+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:06.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:05 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:05.823+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:06.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:06.132+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:06.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:06.132+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:06.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:06.132+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:06.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:06.132+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:06.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:06.132+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:06.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:06.215+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:06.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:06.215+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:06.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:06.215+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:06.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:06.215+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:06.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:06 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:06.215+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.116055+0000 front 2023-11-17T00:08:44.116081+0000 (oldest deadline 2023-11-17T00:09:05.716572+0000) 2023-11-17T00:09:06.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:06 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:06.470+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:06.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:06 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:06.470+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:06.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:06 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:06.470+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:06.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:06 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:06.470+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:06.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:06 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:06.470+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:07.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:06 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:06.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:07.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:06 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:06.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:07.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:06 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:06.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:07.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:06 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:06.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:07.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:06 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:06.794+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:07.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:07.109+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:07.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:07.109+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:07.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:07.109+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:07.109+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:07.109+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:07.240+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.410970+0000 front 2023-11-17T00:08:11.410981+0000 (oldest deadline 2023-11-17T00:08:37.311047+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:07.240+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.012126+0000 front 2023-11-17T00:08:19.012102+0000 (oldest deadline 2023-11-17T00:08:44.912329+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:07.240+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:24.913250+0000 front 2023-11-17T00:08:24.913317+0000 (oldest deadline 2023-11-17T00:08:48.413092+0000) 2023-11-17T00:09:07.608 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:07.240+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.715351+0000 front 2023-11-17T00:08:34.715243+0000 (oldest deadline 2023-11-17T00:08:58.215203+0000) 2023-11-17T00:09:07.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8[148310]: 2023-11-17T00:09:07.240+0000 7fd2c3b15700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.116055+0000 front 2023-11-17T00:08:44.116081+0000 (oldest deadline 2023-11-17T00:09:05.716572+0000) 2023-11-17T00:09:07.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:07.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:07.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:07.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:07.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:07.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:07.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:07.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:07.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:07 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:07.422+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:08.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:08.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:08.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:08.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:08.018 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:08.019 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:07 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:07.761+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:08.433 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:08 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:08.138+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:08.433 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:08 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:08.138+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:08.433 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:08 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:08.138+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:08.434 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:08 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:08.138+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:08.434 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:08 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:08.138+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:08.857 INFO:journalctl@ceph.osd.8.smithi173.stdout:Nov 17 00:09:08 smithi173 bash[162847]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-8 2023-11-17T00:09:08.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:08 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:08.425+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:08.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:08 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:08.425+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:08.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:08 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:08.425+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:08.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:08 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:08.425+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:08.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:08 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:08.425+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:09.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:09.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:09.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:09.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:09.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:09.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:08 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:08.783+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:09.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:09.128+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:09.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:09.128+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:09.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:09.128+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:09.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:09.128+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:09.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:09 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:09.128+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:09.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.8.service' 2023-11-17T00:09:09.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:09.799 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-11-17T00:09:09.799 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-11-17T00:09:09.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.9 2023-11-17T00:09:09.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:09.381+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:09.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:09.382+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:09.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:09.382+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:09.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:09.382+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:09.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:09 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:09.382+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:10.270 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:09.806+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:10.271 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:09 smithi023 systemd[1]: Stopping Ceph osd.9 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:10.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:10.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:10.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:10.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:10.616 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:10.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:10.618 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:10.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:10.618 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:10 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:10.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:10.620 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:10.620 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:10.621 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:10.621 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:10.622 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:10.622 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:10 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:10.344+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:10.768 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.328+0000 7f8314291700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:10.768 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.328+0000 7f8314291700 -1 osd.9 97 *** Got signal Terminated *** 2023-11-17T00:09:10.768 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.328+0000 7f8314291700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:09:11.271 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:11.274 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:11.276 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:11.278 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:11.279 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:11.279 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:10 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:10.830+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:11.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:11 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:11.093+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:11.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:11 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:11.093+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:11.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:11 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:11.093+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:11.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:11 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:11.093+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:11.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:11 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:11.093+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:11.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:11.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:11.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:11.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:11.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:11.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:11 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:11.312+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:12.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:12.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:12.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:12.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:12.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:12.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:11 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:11.854+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:12.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:12.102+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:12.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:12.102+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:12.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:12.102+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:12.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:12.102+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:12.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:12.102+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:12.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:12.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:12.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:12.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:12.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:12.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:12 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:12.340+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:13.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:13.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:13.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:13.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:13.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:13.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:12 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:12.895+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:13.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:13 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:13.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:13.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:13 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:13.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:13.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:13 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:13.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:13.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:13 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:13.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:13.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:13 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:13.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:13.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:13.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:13.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:13.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:13.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:13.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:13 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:13.307+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:14.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:14.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:14.268 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:14.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:14.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:14.269 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:13 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:13.857+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:14.609 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:14 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:14.324+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:14.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:14.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:14.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:14.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:14.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:15.274 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.743698+0000 front 2023-11-17T00:08:02.743581+0000 (oldest deadline 2023-11-17T00:08:28.643796+0000) 2023-11-17T00:09:15.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.144755+0000 front 2023-11-17T00:08:12.144803+0000 (oldest deadline 2023-11-17T00:08:35.045042+0000) 2023-11-17T00:09:15.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:15.045937+0000 front 2023-11-17T00:08:15.045775+0000 (oldest deadline 2023-11-17T00:08:40.945801+0000) 2023-11-17T00:09:15.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.147231+0000 front 2023-11-17T00:08:26.147312+0000 (oldest deadline 2023-11-17T00:08:49.647407+0000) 2023-11-17T00:09:15.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:35.449408+0000 front 2023-11-17T00:08:35.449735+0000 (oldest deadline 2023-11-17T00:09:01.849576+0000) 2023-11-17T00:09:15.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:14 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9[126483]: 2023-11-17T00:09:14.864+0000 7f830e883700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.850331+0000 front 2023-11-17T00:08:41.850342+0000 (oldest deadline 2023-11-17T00:09:07.750036+0000) 2023-11-17T00:09:15.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:15.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:15.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:15.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:15.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:15.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:15 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:15.157+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:15.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:15.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:15.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:15.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:15.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:15.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:15.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:15 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:15.358+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:16.226 INFO:journalctl@ceph.osd.9.smithi023.stdout:Nov 17 00:09:16 smithi023 bash[254988]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-9 2023-11-17T00:09:16.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:16.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:16.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:16.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:16.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:16.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:16 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:16.152+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:16.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:16.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:16.608 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:16.612 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:16.612 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:16.613 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:16.633 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:16 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:16.350+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:17.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:17.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:17.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:17.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:17.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:17.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:17 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:17.201+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:17.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:17.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:17.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:17.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:17.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:17.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:17.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:17 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:17.393+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:17.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.9.service' 2023-11-17T00:09:18.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:18.089 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-11-17T00:09:18.089 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-11-17T00:09:18.089 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.10 2023-11-17T00:09:18.424 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 systemd[1]: Stopping Ceph osd.10 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:18.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:18.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:18.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:18.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:18.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:18.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:18.230+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:18.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:18.949 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.424+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:18.950 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.490+0000 7f27c08bc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:18.950 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.490+0000 7f27c08bc700 -1 osd.10 97 *** Got signal Terminated *** 2023-11-17T00:09:18.950 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:18 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:18.490+0000 7f27c08bc700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:09:19.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:19.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:19.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:19.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:19.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:19.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:19 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:19.213+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:19.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:19.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:19 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:19.434+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:20.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:20.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:20.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:20.611 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:20 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:20.203+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:20.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:20.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:20.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:20.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:20.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:20.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:20.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:20 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:20.440+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:21.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:21.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:21.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:21.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:21.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:21.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:21 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:21.197+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:21.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:21.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:21.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:21.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:21.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:21.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:21.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:21 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:21.435+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:22.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:22.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:22.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:22.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:22.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:22.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:22 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:22.159+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:22.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:22.859 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:22.860 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:22.860 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:22.861 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:22.862 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:22.863 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:22 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:22.463+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:23.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:23.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:23.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:23.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:23.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:23.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:23 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:23.206+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:23.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.568372+0000 front 2023-11-17T00:08:02.568524+0000 (oldest deadline 2023-11-17T00:08:24.868572+0000) 2023-11-17T00:09:23.857 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:11.870196+0000 front 2023-11-17T00:08:11.870312+0000 (oldest deadline 2023-11-17T00:08:37.170408+0000) 2023-11-17T00:09:23.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:20.071519+0000 front 2023-11-17T00:08:20.071498+0000 (oldest deadline 2023-11-17T00:08:46.472136+0000) 2023-11-17T00:09:23.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2023-11-17T00:08:26.472672+0000 front 2023-11-17T00:08:26.472586+0000 (oldest deadline 2023-11-17T00:08:50.572670+0000) 2023-11-17T00:09:23.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:34.073727+0000 front 2023-11-17T00:08:34.073758+0000 (oldest deadline 2023-11-17T00:08:59.973979+0000) 2023-11-17T00:09:23.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:44.075242+0000 front 2023-11-17T00:08:44.075415+0000 (oldest deadline 2023-11-17T00:09:09.975081+0000) 2023-11-17T00:09:23.858 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:23 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10[152662]: 2023-11-17T00:09:23.451+0000 7f27b8eaa700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.676341+0000 front 2023-11-17T00:08:51.676386+0000 (oldest deadline 2023-11-17T00:09:13.376189+0000) 2023-11-17T00:09:24.230 INFO:journalctl@ceph.osd.10.smithi156.stdout:Nov 17 00:09:24 smithi156 bash[161645]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-10 2023-11-17T00:09:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:24.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:24.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:24.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:24 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:24.221+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:25.596 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.10.service' 2023-11-17T00:09:25.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:25.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:25.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:25.608 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:25.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:25.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.182+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:25.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:25.636 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-11-17T00:09:25.636 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-11-17T00:09:25.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.11 2023-11-17T00:09:25.981 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 systemd[1]: Stopping Ceph osd.11 for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:26.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.979+0000 7fc6c9179700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:26.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.979+0000 7fc6c9179700 -1 osd.11 97 *** Got signal Terminated *** 2023-11-17T00:09:26.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:25 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:25.979+0000 7fc6c9179700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:26.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:26 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:26.135+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2023-11-17T00:09:00.161189+0000 front 2023-11-17T00:09:00.161282+0000 (oldest deadline 2023-11-17T00:09:26.061022+0000) 2023-11-17T00:09:27.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:27.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:27.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:27.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:27.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:27.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:27.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:27 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:27.104+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2023-11-17T00:09:00.161189+0000 front 2023-11-17T00:09:00.161282+0000 (oldest deadline 2023-11-17T00:09:26.061022+0000) 2023-11-17T00:09:28.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.096+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:28.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.096+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:28.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.096+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:28.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.097+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:28.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.097+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:28.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.097+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:28.358 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:28 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:28.097+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2023-11-17T00:09:00.161189+0000 front 2023-11-17T00:09:00.161282+0000 (oldest deadline 2023-11-17T00:09:26.061022+0000) 2023-11-17T00:09:29.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:29.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:29.612 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:29.613 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:29.613 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:29.613 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:29.613 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:29 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:29.141+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2023-11-17T00:09:00.161189+0000 front 2023-11-17T00:09:00.161282+0000 (oldest deadline 2023-11-17T00:09:26.061022+0000) 2023-11-17T00:09:30.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2023-11-17T00:08:02.949442+0000 front 2023-11-17T00:08:02.949313+0000 (oldest deadline 2023-11-17T00:08:25.749862+0000) 2023-11-17T00:09:30.485 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2023-11-17T00:08:12.052074+0000 front 2023-11-17T00:08:12.052092+0000 (oldest deadline 2023-11-17T00:08:35.552037+0000) 2023-11-17T00:09:30.485 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2023-11-17T00:08:19.454381+0000 front 2023-11-17T00:08:19.454404+0000 (oldest deadline 2023-11-17T00:08:42.354543+0000) 2023-11-17T00:09:30.485 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2023-11-17T00:08:33.957184+0000 front 2023-11-17T00:08:33.957209+0000 (oldest deadline 2023-11-17T00:08:58.657038+0000) 2023-11-17T00:09:30.486 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2023-11-17T00:08:41.458496+0000 front 2023-11-17T00:08:41.458515+0000 (oldest deadline 2023-11-17T00:09:04.958633+0000) 2023-11-17T00:09:30.486 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2023-11-17T00:08:51.360317+0000 front 2023-11-17T00:08:51.360218+0000 (oldest deadline 2023-11-17T00:09:14.260049+0000) 2023-11-17T00:09:30.486 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2023-11-17T00:09:00.161189+0000 front 2023-11-17T00:09:00.161282+0000 (oldest deadline 2023-11-17T00:09:26.061022+0000) 2023-11-17T00:09:30.487 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:30 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11[153749]: 2023-11-17T00:09:30.092+0000 7fc6c1f68700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2023-11-17T00:09:06.061407+0000 front 2023-11-17T00:09:06.061421+0000 (oldest deadline 2023-11-17T00:09:29.561484+0000) 2023-11-17T00:09:31.858 INFO:journalctl@ceph.osd.11.smithi173.stdout:Nov 17 00:09:31 smithi173 bash[163117]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-osd-11 2023-11-17T00:09:32.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@osd.11.service' 2023-11-17T00:09:32.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:32.970 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-11-17T00:09:32.970 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-11-17T00:09:32.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.a 2023-11-17T00:09:33.494 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 17 00:09:33 smithi023 systemd[1]: Stopping Ceph mds.a for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:33.768 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 17 00:09:33 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-a[131818]: 2023-11-17T00:09:33.491+0000 7fa65bd2f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:33.768 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 17 00:09:33 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-a[131818]: 2023-11-17T00:09:33.491+0000 7fa65bd2f700 -1 mds.a *** got signal Terminated *** 2023-11-17T00:09:35.266 INFO:journalctl@ceph.mds.a.smithi023.stdout:Nov 17 00:09:35 smithi023 bash[255254]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-a 2023-11-17T00:09:35.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.a.service' 2023-11-17T00:09:35.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:35.673 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-11-17T00:09:35.673 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-11-17T00:09:35.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.d 2023-11-17T00:09:36.018 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 17 00:09:35 smithi023 systemd[1]: Stopping Ceph mds.d for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:36.518 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 17 00:09:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-d[130814]: 2023-11-17T00:09:36.174+0000 7f02a6bf7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:36.519 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 17 00:09:36 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-d[130814]: 2023-11-17T00:09:36.174+0000 7f02a6bf7700 -1 mds.d *** got signal Terminated *** 2023-11-17T00:09:40.769 INFO:journalctl@ceph.mds.d.smithi023.stdout:Nov 17 00:09:40 smithi023 bash[255393]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-d 2023-11-17T00:09:40.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.d.service' 2023-11-17T00:09:40.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:40.985 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-11-17T00:09:40.985 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-11-17T00:09:40.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.g 2023-11-17T00:09:41.268 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 17 00:09:41 smithi023 systemd[1]: Stopping Ceph mds.g for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:41.768 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 17 00:09:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-g[131430]: 2023-11-17T00:09:41.490+0000 7f3be6614700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:41.769 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 17 00:09:41 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-g[131430]: 2023-11-17T00:09:41.490+0000 7f3be6614700 -1 mds.g *** got signal Terminated *** 2023-11-17T00:09:44.268 INFO:journalctl@ceph.mds.g.smithi023.stdout:Nov 17 00:09:43 smithi023 bash[255536]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-g 2023-11-17T00:09:44.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.g.service' 2023-11-17T00:09:44.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:44.470 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-11-17T00:09:44.470 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-11-17T00:09:44.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.j 2023-11-17T00:09:44.768 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 17 00:09:44 smithi023 systemd[1]: Stopping Ceph mds.j for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:45.268 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 17 00:09:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-j[130426]: 2023-11-17T00:09:44.980+0000 7fa1b94a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:45.268 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 17 00:09:44 smithi023 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-j[130426]: 2023-11-17T00:09:44.980+0000 7fa1b94a6700 -1 mds.j *** got signal Terminated *** 2023-11-17T00:09:49.519 INFO:journalctl@ceph.mds.j.smithi023.stdout:Nov 17 00:09:49 smithi023 bash[255679]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-j 2023-11-17T00:09:49.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.j.service' 2023-11-17T00:09:49.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:49.703 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-11-17T00:09:49.704 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-11-17T00:09:49.704 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.b 2023-11-17T00:09:50.062 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:09:49 smithi156 systemd[1]: Stopping Ceph mds.b for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:50.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:09:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: 2023-11-17T00:09:50.060+0000 7fa8ad3ae700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:50.357 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:09:50 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b[156429]: 2023-11-17T00:09:50.060+0000 7fa8ad3ae700 -1 mds.b *** got signal Terminated *** 2023-11-17T00:09:54.331 INFO:journalctl@ceph.mds.b.smithi156.stdout:Nov 17 00:09:54 smithi156 bash[161907]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-b 2023-11-17T00:09:54.515 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.b.service' 2023-11-17T00:09:54.552 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:54.552 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-11-17T00:09:54.552 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-11-17T00:09:54.553 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.e 2023-11-17T00:09:54.857 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:09:54 smithi156 systemd[1]: Stopping Ceph mds.e for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:55.357 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:09:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: 2023-11-17T00:09:54.925+0000 7f876ac62700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:55.357 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:09:54 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e[155747]: 2023-11-17T00:09:54.925+0000 7f876ac62700 -1 mds.e *** got signal Terminated *** 2023-11-17T00:09:58.607 INFO:journalctl@ceph.mds.e.smithi156.stdout:Nov 17 00:09:58 smithi156 bash[162052]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-e 2023-11-17T00:09:58.629 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.e.service' 2023-11-17T00:09:58.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:09:58.665 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-11-17T00:09:58.665 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-11-17T00:09:58.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.h 2023-11-17T00:09:59.035 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:09:58 smithi156 systemd[1]: Stopping Ceph mds.h for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:09:59.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:09:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: 2023-11-17T00:09:59.034+0000 7fcd5abf8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:09:59.357 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:09:59 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h[156091]: 2023-11-17T00:09:59.034+0000 7fcd5abf8700 -1 mds.h *** got signal Terminated *** 2023-11-17T00:10:02.607 INFO:journalctl@ceph.mds.h.smithi156.stdout:Nov 17 00:10:02 smithi156 bash[162198]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-h 2023-11-17T00:10:02.623 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.h.service' 2023-11-17T00:10:02.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:02.662 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-11-17T00:10:02.662 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-11-17T00:10:02.662 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.k 2023-11-17T00:10:03.043 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 17 00:10:02 smithi156 systemd[1]: Stopping Ceph mds.k for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:10:03.357 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 17 00:10:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-k[155408]: 2023-11-17T00:10:03.041+0000 7f5d214ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:10:03.357 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 17 00:10:03 smithi156 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-k[155408]: 2023-11-17T00:10:03.041+0000 7f5d214ec700 -1 mds.k *** got signal Terminated *** 2023-11-17T00:10:07.269 INFO:journalctl@ceph.mds.k.smithi156.stdout:Nov 17 00:10:06 smithi156 bash[162358]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-k 2023-11-17T00:10:07.275 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.k.service' 2023-11-17T00:10:07.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:07.309 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-11-17T00:10:07.309 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-11-17T00:10:07.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c 2023-11-17T00:10:07.607 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:10:07 smithi173 systemd[1]: Stopping Ceph mds.c for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:10:08.107 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:10:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[160342]: 2023-11-17T00:10:07.665+0000 7fa751be3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:10:08.107 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:10:07 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c[160342]: 2023-11-17T00:10:07.665+0000 7fa751be3700 -1 mds.c *** got signal Terminated *** 2023-11-17T00:10:12.107 INFO:journalctl@ceph.mds.c.smithi173.stdout:Nov 17 00:10:11 smithi173 bash[163393]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-c 2023-11-17T00:10:12.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.c.service' 2023-11-17T00:10:12.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:12.204 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-11-17T00:10:12.204 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-11-17T00:10:12.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.f 2023-11-17T00:10:12.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:10:12 smithi173 systemd[1]: Stopping Ceph mds.f for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:10:12.857 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:10:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: 2023-11-17T00:10:12.605+0000 7f1e5b7af700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:10:12.857 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:10:12 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f[157016]: 2023-11-17T00:10:12.605+0000 7f1e5b7af700 -1 mds.f *** got signal Terminated *** 2023-11-17T00:10:14.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:Nov 17 00:10:14 smithi173 bash[163537]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-f 2023-11-17T00:10:14.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.f.service' 2023-11-17T00:10:14.507 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:14.510 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-11-17T00:10:14.513 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-11-17T00:10:14.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.i 2023-11-17T00:10:14.857 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:10:14 smithi173 systemd[1]: Stopping Ceph mds.i for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:10:15.359 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:10:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: 2023-11-17T00:10:14.927+0000 7fef27a7b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:10:15.359 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:10:14 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i[157698]: 2023-11-17T00:10:14.927+0000 7fef27a7b700 -1 mds.i *** got signal Terminated *** 2023-11-17T00:10:18.351 INFO:journalctl@ceph.mds.i.smithi173.stdout:Nov 17 00:10:18 smithi173 bash[163677]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-i 2023-11-17T00:10:18.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.i.service' 2023-11-17T00:10:18.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:18.569 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-11-17T00:10:18.569 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-11-17T00:10:18.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l 2023-11-17T00:10:18.858 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:10:18 smithi173 systemd[1]: Stopping Ceph mds.l for d04064ea-84da-11ee-8dbe-212e2dc638e7... 2023-11-17T00:10:19.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:10:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: 2023-11-17T00:10:18.937+0000 7f07c0833700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-17T00:10:19.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:10:18 smithi173 ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l[160656]: 2023-11-17T00:10:18.937+0000 7f07c0833700 -1 mds.l *** got signal Terminated *** 2023-11-17T00:10:22.312 INFO:journalctl@ceph.mds.l.smithi173.stdout:Nov 17 00:10:22 smithi173 bash[163814]: ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7-mds-l 2023-11-17T00:10:22.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d04064ea-84da-11ee-8dbe-212e2dc638e7@mds.l.service' 2023-11-17T00:10:22.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T00:10:22.546 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-11-17T00:10:22.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-17T00:10:27.052 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-17T00:10:37.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-17T00:10:39.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:10:39.886 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:10:39.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T00:10:39.951 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-17T00:10:39.954 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi023/crash 2023-11-17T00:10:39.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T00:10:40.034 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T00:10:40.035 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2023-11-17T00:10:40.038 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi156/crash 2023-11-17T00:10:40.039 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T00:10:40.108 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T00:10:40.108 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2023-11-17T00:10:40.114 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi173/crash 2023-11-17T00:10:40.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T00:10:40.164 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T00:10:40.164 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2023-11-17T00:10:40.165 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-17T00:10:40.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-17T00:10:40.244 INFO:teuthology.orchestra.run.smithi023.stderr:grep: /var/log/ceph/d04064ea-84da-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-17T00:10:40.248 INFO:tasks.cephadm:Compressing logs... 2023-11-17T00:10:40.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:10:40.253 DEBUG:teuthology.orchestra.run.smithi156:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:10:40.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:10:40.285 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-17T00:10:40.286 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-17T00:10:40.318 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-17T00:13:52.010 INFO:tasks.cephadm:Archiving logs... 2023-11-17T00:13:52.011 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi023/log 2023-11-17T00:13:52.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T00:14:00.398 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi156/log 2023-11-17T00:14:00.399 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T00:14:17.702 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi173/log 2023-11-17T00:14:17.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T00:14:41.699 INFO:tasks.cephadm:Removing cluster... 2023-11-17T00:14:41.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force 2023-11-17T00:14:43.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force 2023-11-17T00:14:44.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d04064ea-84da-11ee-8dbe-212e2dc638e7 --force 2023-11-17T00:14:45.168 INFO:tasks.cephadm:Removing cephadm ... 2023-11-17T00:14:45.168 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T00:14:45.221 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T00:14:45.247 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T00:14:45.269 INFO:tasks.cephadm:Teardown complete 2023-11-17T00:14:45.269 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-17T00:14:45.282 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-17T00:14:45.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-17T00:14:45.287 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-17T00:14:45.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-17T00:14:45.468 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-17T00:14:45.481 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-17T00:14:45.482 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T00:14:45.487 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T00:14:45.503 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T00:14:45.519 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2023-11-17T00:14:45.520 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-11-17T00:14:45.523 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T00:14:45.523 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2023-11-17T00:14:45.523 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +1052us[+1068us] +/- 82ms 2023-11-17T00:14:45.523 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -849us[ -833us] +/- 62ms 2023-11-17T00:14:45.523 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +72us[ +72us] +/- 74ms 2023-11-17T00:14:45.524 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T00:14:45.525 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T00:14:45.525 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-11-17T00:14:45.525 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +1050us[+1050us] +/- 82ms 2023-11-17T00:14:45.526 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -879us[ -880us] +/- 62ms 2023-11-17T00:14:45.526 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +77us[ +77us] +/- 74ms 2023-11-17T00:14:45.526 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T00:14:45.526 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-11-17T00:14:45.545 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T00:14:45.545 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-11-17T00:14:45.545 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +1049us[+1076us] +/- 81ms 2023-11-17T00:14:45.546 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -830us[ -796us] +/- 62ms 2023-11-17T00:14:45.546 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +87us[ +114us] +/- 73ms 2023-11-17T00:14:45.546 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T00:14:45.547 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-17T00:14:45.570 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-17T00:14:45.573 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-17T00:14:45.594 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-17T00:14:45.868 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-17T00:14:45.971 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-17T00:14:46.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T00:14:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:46.148 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T00:14:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:46.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T00:14:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:46.220 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-17T00:14:46.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-17T00:14:46.250 INFO:teuthology.task.internal:Duration was 2462.122468 seconds 2023-11-17T00:14:46.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-17T00:14:46.265 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-17T00:14:46.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T00:14:46.271 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T00:14:46.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T00:14:46.320 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T00:14:46.321 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T00:14:46.452 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T00:14:46.828 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-17T00:14:46.828 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2023-11-17T00:14:46.828 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T00:14:46.913 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-11-17T00:14:46.914 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T00:14:46.939 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2023-11-17T00:14:46.939 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T00:14:46.963 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-17T00:14:46.963 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:14:46.968 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:14:46.986 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T00:14:47.111 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-17T00:14:47.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T00:14:47.117 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T00:14:47.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T00:14:47.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-17T00:14:47.336 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-17T00:14:47.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T00:14:47.415 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T00:14:47.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T00:14:47.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-17T00:14:47.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.495 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.522 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-11-17T00:14:47.536 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2023-11-17T00:14:47.576 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2023-11-17T00:14:47.615 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:47.658 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:47.674 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T00:14:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T00:14:47.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-17T00:14:47.814 INFO:teuthology.task.internal:Transferring archived files... 2023-11-17T00:14:47.848 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi023 2023-11-17T00:14:47.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T00:14:48.036 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi156 2023-11-17T00:14:48.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T00:14:48.142 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178/remote/smithi173 2023-11-17T00:14:48.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T00:14:48.386 INFO:teuthology.task.internal:Removing archive directory... 2023-11-17T00:14:48.386 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T00:14:48.392 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T00:14:48.397 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T00:14:48.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-17T00:14:48.447 INFO:teuthology.task.internal:Not uploading archives. 2023-11-17T00:14:48.448 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-17T00:14:48.472 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-17T00:14:48.472 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T00:14:48.478 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T00:14:48.480 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T00:14:48.495 INFO:teuthology.orchestra.run.smithi156.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 00:14 /home/ubuntu/cephtest 2023-11-17T00:14:48.496 INFO:teuthology.orchestra.run.smithi173.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 00:14 /home/ubuntu/cephtest 2023-11-17T00:14:48.519 INFO:teuthology.orchestra.run.smithi023.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 00:14 /home/ubuntu/cephtest 2023-11-17T00:14:48.521 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-17T00:14:48.532 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-17T00:14:48.587 INFO:teuthology.nuke:Checking targets against current locks 2023-11-17T00:14:48.619 DEBUG:teuthology.nuke:shortname: smithi023 2023-11-17T00:14:48.620 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T00:14:48.646 DEBUG:teuthology.nuke:shortname: smithi156 2023-11-17T00:14:48.647 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T00:14:48.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.704348', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-17T00:14:48.675 DEBUG:teuthology.nuke:shortname: smithi173 2023-11-17T00:14:48.675 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T00:14:48.681 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.708576', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-17T00:14:48.715 INFO:teuthology.orchestra.console.smithi023:Power off 2023-11-17T00:14:48.716 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T00:14:48.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_21:22:00-fs-wip-choffman-62793-distro-default-smithi/7461178', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 23:23:23.706881', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-17T00:14:48.742 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2023-11-17T00:14:48.742 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:48.771 INFO:teuthology.orchestra.console.smithi156:Power off 2023-11-17T00:14:48.771 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T00:14:48.787 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2023-11-17T00:14:48.801 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2023-11-17T00:14:48.801 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:48.827 INFO:teuthology.orchestra.console.smithi173:Power off 2023-11-17T00:14:48.828 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T00:14:48.839 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-17T00:14:48.860 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2023-11-17T00:14:48.861 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:48.905 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-11-17T00:14:52.788 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:52.840 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:52.901 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2023-11-17T00:14:52.906 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:14:52.951 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-17T00:14:53.018 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-11-17T00:14:56.905 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:15:08.825 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:15:08.834 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T00:15:08.927 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2023-11-17T00:15:08.936 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2023-11-17T00:15:08.944 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2023-11-17T00:15:09.028 INFO:teuthology.orchestra.console.smithi023:Power off completed 2023-11-17T00:15:09.037 INFO:teuthology.orchestra.console.smithi156:Power off completed 2023-11-17T00:15:09.044 INFO:teuthology.orchestra.console.smithi173:Power off completed 2023-11-17T00:15:09.182 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5b73517e0e2fa132ea79290e5fc06ea5349b2288 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} duration: 2462.1224682331085 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi023 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c9565146c8647d4e00eb6c7ae626dc192680e7b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=30ea35db2aab44929baa459c1f2ed37c status: fail success: false 2023-11-17T00:15:09.182 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T00:15:09.259 INFO:teuthology.run:FAIL